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1.0  INTRODUCTION 


1.1  BACKGROUND 

The  cost  effectiveness  of  fault  detection  and  isolation  techniques  used  in  electronic 
systems  is  a  present  concern  of  the  military,  now  that  testing  exceeds  25%  of  the  total 
life  cycle  cost  of  a  system  (1).  Problems  include  assuring  adequate  failure  coverage  by 
Built-In  Test  and  Automatic  Test  Equipment  (BIT/ATE),  reducing  false  alarms,  isolating 
intermittent  failures,  and  reducing  maintenance  training  requirements. 

1.2  OBJECTIVES 

This  study  provides  the  foundation  for  a  logical  and  cost-effective  program  for  applying 
Artificial  Intelligence  to  testability.  Emphasis  is  on  those  artificial  intelligence  (AI) 
techniques  capable  of  practical  application  to  testability  problems  immediately  or  with 
low  risk  development  within  three  to  five  years. 

1.3  SCOPE  OF  STUDY 

This  study  surveyed  AI  techniques  for  potential  application  to  electronic  testability. 
Fault  detection  and  fault  isolation  are  primarily  accomplished  in  military  electronic 
systems  with  Built-in  Test  (BIT)  and  Automatic  Test  Equipment  (ATE).  The  extent  of 
current  use  of  AI  in  BIT  and  ATE  was  identified.  The  current  problems  in  testability  were 
identified,  potential  AI  solutions  were  analyzed  and  near  term  opportunities  were 
evaluated. 

The  survey  considered  a  wide  variety  of  military  electronics  and  was  concerned  with  use 
of  artificial  intelligence  techniques  that  could  provide  improvements  to  functional  and 
diagnostic  testing  of  systems,  boxes,  and  cards  that  must  operate  in  a  wide  range  of 
environments.  That  equipment  must  generally  be  very  reliable  while  operating  under 
rugged  environmental  conditions  since  the  liability  of  malfunction  is  often  of  military  or 
life  importance.  The  electronics  costs  are  very  high  while  production  volume  is  typically 
low.  It  should  be  noted  that  testability  concerns  for  commercial  systems  are  different  for 
a  majority  of  these  characteristics.  The  largest  differences  are  in  production  quantity 
and  use  environment.  This  study  addresses  only  the  military  applications  of  Artificial 
Intelligence  to  testability 
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1.4  REPORT  ORGANIZATION 

The  following  sections  describe  the  tasks  performed  for  this  study  and  briefly  summarize 
the  findings.  Each  subsection  corresponds  to  a  major  section  of  this  report. 

l.t.1  Survey  of  AI  Techniques 

A  survey  was  made  of  existing  and  developing  AI  techniques  by  literature  search  and 
interviews  with  national  AI-Testability  experts.  A  summary  of  the  various  AI  techniques 
investigated  and  their  applicability  to  testability  are  provided  in  section  2.0.  The 

bibliography  and  industry  survey  appendices  contain  the  details  of  all  parts  of  the 
survey. 

The  AI  techniques  survey  included  all  of  the  standard  AI  techniques,  of  which  the 
following  were  found  to  have  near  term  application  to  testability: 

o  Expert  Consulting  Systems 

o  Intelligent  Retrieval  from  Data  Base 
o  Perception 

o  Machine  Learning 

A  restricted  version  of  Machine  Learning  was  added  to  the  initial  list  as  it  proved  to  be  a 
very  cost  effective  way  of  creating  types  of  expert  systems  for  problems  which  can  be 
expressed  in  a  tabular  format,  as  is  true  for  many  maintenance  problems . 

The  other  techniques  either  were  not  well  enough  developed,  did  not  have  any  application, 
would  not  produce  a  cost  effective  application,  or  were  not  appropriate  for  low  volume 
military  electronics.  These  included: 

o  Theorem  Proving 

o  Natural  Language  Processing 

o  Robotics 

o  Automatic  Programming 

o  Modeling 

o  Computer-Aided  Instruction 
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1.4.2  Survey  of  Existing  and  Developing  ATE/BIT 

The  two  primary  methods  of  testing  military  electronics  are  Built-In  Test  (BIT),  and 
Automatic  Test  Equipment  (ATE).  The  BIT  portion  of  military  electronics  adds  2%  to  20% 
additional  hardware  to  confirm  the  functioning  of  90%  to  100%  of  the  electronics  and 
sometimes  provide  fault  isolation.  ATE  is  used  for  off-line,  out  of  context  test  of  a  unit, 
generally  not  performed  in  the  field. 

Many  possible  AI  applications  were  found  for  these  two  areas,  but  only  a  few  were  found 
to  be  of  low  enough  risk  and  high  enough  benefit  to  warrant  military  R<5cD  activity.  The 
results  of  the  ATE/BIT  survey,  in  section  3.0,  showed  it  to  be  much  more  cost  effective  to 
use  AI  techniques  to  eliminate  the  source  of  a  testability  problem  (in  the  box  or  system) 
than  to  focus  on  the  resulting  symptom  (i.e.,  better  diagnosis  tools  or  better  ATE).  The 
initial  evaluations  of  the  AI  applications  are  documented  in  a  matrix  in  section  5.1. 
Extensive  references  are  contained  in  the  bibliography  and  industry  appendices . 

1.4.3  Survey  of  Testability  Applications 

This  applications  survey  included  the  following  six  possible  testability  improvement  areas 
with  a  potential  for  near  term  application  with,  or  as  part  of,  an  AI  approach: 

o  Self  improving  diagnostics 

o  More  effective  fault  detection  and  isolation 

o  Discrimination  between  false  alarms  and  intermittent  faults 
o  Reduction  of  skills  required  for  maintenance 
o  Integrated  diagnostics 

o  Design  for  testability 

Evolutionary  progress  is  being  made  in  these  areas.  AI  provides  tools  which  can  speed  up 
this  evolution  by  improving  information  management. 

1.4.4  Potential  AI  Solutions  to  Problems 

Section  5.0  discusses  the  initial  AI  application  analysis  for  military  needs.  Current 
testability  problems  were  identified  and  potential  AI  applications  to  those  problems 
evaluated.  The  following  testability  applications  of  Artificial  Intelligence  were  specifi¬ 
cally  identified  for  investigation: 
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o  Computer-Aided  Preliminary  Design  for  Testability 
o  Smart  Built-In  Test 

o  Smart  System  Integrated  Test 

o  Maintenance  Expert  -  Box  Level 

o  Maintenance  Expert  -  System  Level 

o  Maintenance  Expert  -  System  Level  with  Smart  Bit/Self  Improving  Diagnostics 
o  Analog  Automatic  Test  Program  Generation 

o  Smart  Bench  (Computer  Aided  Bench  Testing  that  learns  diagnostics) 
o  Digital  Automatic  Test  Program  Generation  with  Machine  Learning 

It  was  found  that  ail  of  these  application  opportunities  could  be  used  with  or  developed  by 
engineering  work  stations  (EWS),  which  are  becoming  available  directly  to  designers.  EWS 
would  allow  low  cost  development  and  widespread  use.  Detailed  discussions  that  typically 
include  charts  of  evolutionary  development  are  presented  for  each  opportunity. 

1.4.5  Final  Evaluation 

A  ranking  of  AI  opportunities  was  made  based  on  seven  criteria  for  life  cycle  cost  (LCC) 
impact  as  well  nine  other  risk/cost  and  benefits  associated  with  the  development  of  each. 

1.4.6  Study  Findings  and  Recommendations 

The  primary  message  in  section  7.0  is  that  curing  testability  problems,  not  just  their 
symptoms,  is  a  goal  that  is  common  to  all  the  applications  investigated  in  section  5.0,  and 
that  AI  can  provide  hardware  and  software  tools  necessary  to  improve  information 
capture,  flow  and  processing.  A  more  detailed  presentation  of  study  findings  and 
recommendations  is  also  included  in  that  section.  The  primary  recommendation  made  is 
for  a  phased  evolution  of  several  applications  that  emphasizes  common  support  hardware 
and  software  for  developers  and  end  users. 

1.4.7  References 

The  documents  specifically  referenced  by  the  report  are  listed  in  section  8.1.  Other 
reference  material  which  contributed  to  this  study  is  listed  in  the  annotated  bibliography 
of  section  8.2.4  which  is  organized  by  subject  (i.e.,  BIT/ATE,  Maintenance  Experts)  and 
contains  material  of  interest  to  readers  who  want  a  broader  set  of  recent  information  on 
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AI  and  testability.  Items  found  by  the  survey  which  were  several  years  old  or  only  of 
theoretical  interest  are  not  included. 

A  list  of  the  people  interviewed  is  provided  in  section  8.3.  The  recommendations  of  four 
relevant  test  studies  in  1983  provided  in  section  8.3  give  a  perspective  from  which  to  view 
the  recommendations  of  this  study. 


2.0  SURVEY  OF  AI  TECHNIQUES 


A  survey  was  made  of  eleven  AI  techniques  in  order  to  judge  their  potential  applicability 
to  testability  when  used  singly  or  in  combinations.  A  wide  variation  was  found  in  the 
maturity  of  development  between  the  techniques.  The  findings,  summarized  below,  show 
the  wide  variance  in  potential  between  the  techniques.  The  specific,  numbered 
references  in  this  document  can  be  found  in  section  8.1.  Many  other  relevant  references, 
not  specifically  addressed  in  this  section  but  lending  support  to  our  findings,  are  found  in 
the  annotated  bibliography,  section  8.2.  The  conclusions  reached  as  a  result  of  the  AI 
survey  are  in  section  2.12. 

THEOREM  PROVING:  There  is  currently  no  active  work  in  this  area,  and  no  direct  need 
is  seen  for  its  use  in  testability  applications  that  are  not  now  being  solved  by  other 
means. 

NATURAL  LANGUAGE  PROCESSING  is  a  very  active,  but  as  yet  immature,  area.  No 
substantial  cost  savings  could  be  expected  to  result  from  testability  applications,  but 
worker  productivity  would  increase. 

INTELLIGENT  RETRIEVAL  FROM  DATA  BASES  is  being  developed  by  more  than  30 
groups,  and  there  should  be  substantial  incremental  development  during  the  next  few 
years.  This  technique  improves  testability  by  improving  efficiency  of  data  base  search 
operations. 

EXPERT  CONSULTING  SYSTEMS  is  the  hottest  AI  topic  with  a  number  of  commercial 
successes  demonstrated  and  many  military  developments  are  currently  underway.  Main¬ 
tenance  expert  systems  are  expensive  to  implement,  but  there  appear  to  be  approaches 
that  avoid  the  knowledge  engineering  bottleneck.  Metarules  and  machine  learning  have 
the  potential  to  substantially  reduce  costs  and  development  time.  Expert  systems  with 
ability  to  understand  circuit  function  and  malfunction  are  not  expected  for  at  least 
5  years. 

ROBOTICS  applications  to  factory  production  is  a  developing  new  field.  There  is  little 
economic  justification  for  near  term  use  of  robotics  for  low  volume  production  or 
maintenance. 


AUTOMATIC  PROGRAMMING  may  have  some  use  in  test  program  generation,  but  this  is 
unproven  even  for  the  commercial  market. 

PERCEPTION  and  pattern  recognition  for  discrimination  of  faise  alarm  and  intermit* 
tents  will  be  viable  in  3  to  5  years.  Dr.  Pau's  work  shows  the  current  capability. 

COMBINATORIAL  AND  SCHEDULING  PROBLEMS  have  no  current  application  to 
testability.  Needs  in  this  area  are  already  covered  by  other  techniques  (i.e.  operations 
research). 

VOICE  SYNTHESIS  AND  RECOGNITION  is  very  active  but  computer  generated  graphics 
should  provide  more  benefits  in  the  near  term. 

INTELLIGENT  COMPUTER  AIDED  INSTRUCTION  is  being  actively  investigated  by 
military  training  organizations.  It  provides  no  cost  effective  benefits  to  testability  in 
the  near  term. 

MACHINE  LEARNING  can  provide  a  breakthrough  in  reducing  development  costs  for 
some  maintenance  expert  systems. 


2.1  THEOREM  PROVING 


Our  survey  for  this  contract  found  no  evidence  to  change  the  following  statement  by 
Boeing  in  the  Integrated  Testing  and  Maintenance  Techniques  (ITMC1))  final  report: 


Theorem  proving— this  branch  of  AI  attempts  to  produce  programs  that  can  use  a 
given  logical  concept  to  prove  new  ones.  This  is  not  immediately  applicable  to  the 
testing  and  maintenance  field  because  the  real  world  concepts  and  rules  are  not  well 
defined  and  the  input  data  is  noisy.  Current  theorem  proving  systems  using 
predicate  calculus  and  propositional  logic  are  not  readily  adaptable  to  this  scenario. 
Future  developments  in  the  field  of  fuzzy  logic  may  make  theorem  proving  more 
applicable." 

The  Artificial  Intelligence  Applications  to  Maintenance  Study  (AIAM  (3))  suggested 
possible  applications  of  theorem  proving,  but  no  one  has  tried  a  practical  application  of 
this  theoretical  concept: 

"BIT  signals  are  too  often  false  alarms.  In  addition,  intermittent  failures  are 
difficult  to  distinguish  from  false  alarms.  Theorem  proving  techniques  could  be 
employed  to  test  BIT  signals  to  discriminate  against  false  alarms  and  to  identify 
intermittent  failures." 

2.2  NATURAL  LANGUAGE  PROCESSING 


Natural  language  processing  is  the  branch  of  AI  that  attempts  to  create  machines  with 
human-like  communication  skills.  This  field  must  borrow  heavily  from  linguistics  and 
other  sciences  to  understand  how  humans  communicate. 


Natural  language  processing  programs  are  usually  used  as  an  interface  between  a 
nontechnical  system  operator  and  a  complex  computer  system.  For  example,  a  natural 
language  interface  could  be  used  as  an  interface  to  an  intelligent  data  retrieval  system. 


AIAM  (3)  stated: 

1  "A  medical  diagnosis  system,  MYCIN,  can  carry  on  a  conversation  with  a  doctor 

about  bacterial  infections.  It  is  certainly  a  near  term  possibility  that  a  maintenance 
system  can  be  made  to  converse  with  its  user  in  a  useful  subset  of  English.  This 
would  be  accomplished  by  the  user  typing  in  his  side  of  the  conversation.  The 
machine  could  respond  by  printout,  CRT  display  or  synthesized  speech." 

I 
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"Spoken  Interaction  would  be  highly  desirable  as  it  would  free  the  maintenance  man 
from  the  terminal.  He  could  then  have  both  hands  free  and  could  operate  in  remote 
locations  using  a  portable  headset  to  consult  his  computer." 

The  Personal  Electronic  Aid  to  Maintenance  (23)  is  a  portable  unit  with  some  speech  input 
capability  that  addresses  this  last  item.  The  NBS  report  "An  Overview  of  Computer* 
Based  Natural  Language  Processing"  (NBS1R  S3-26S7)  is  an  excellent  70  page  summary  of 
the  issues,  research  required,  forecast,  etc. 


Overall,  while  natural  language  processing  can  be  considered  a  developing  area,  with 
potential  for  application  to  testability  problems  (i.e.  speech  input/response  units),  it  has 
not  matured  enough  for  immediate  application.  Limited  (if  expensive)  applicability  is 
possible  in  3  to  5  years  as  speech  recognition  units  increase  their  vocabulary  and  reduce  or 
eliminate  their  training  requirement  to  adapt  to  a  user's  voice.  Voice  response  units, 
while  more  mature  still  require  substantial  resources  for  random  statement  generation 
capability;  voice  quality  is  improving  and  is  now  well  beyond  the  mechanical,  monotone 
computer  voice  that  has  become  a  cliche  of  synthesized  speech  (see  also  section  2.9). 


2.3  INTELLIGENT  RETRIEVAL  PROM  DATA  BASE 


There  are  many  testability  applications  that  must  retrieve  information  from  a  database. 
An  intelligent  retrieval  speeds  up  the  process  and  increases  the  amount  of  relevant  data 
found.  The  ITM  description  was: 


"Intelligent  retrieval  from  data  base— a  means  by  which  a  maintenance  crewmember 
can  acquire  a  faster  understanding  of  fault  characteristics.  The  retrieval  scheme 
might,  for  instance,  contain  a  set  of  simple  correlation  pairs  that  are  analyzed 
during  postflight.  These  pairs  might  check  a  given  type  of  fault  against  flight  mode, 
acceleration,  temperature,  etc.,  and  the  system  would  notify  the  crewmember  of 
any  recognized  correlations.  Additionally,  the  system  could  contain  more  complex 
correlations  such  as,  if  conditions  A,  B,  and  C  occur  within  10  min,  then  fault  X 
probably  is  causing  the  failures."  (1) 

AIAM  (3)  had  a  different  application  in  mind: 

"Inferential  Retrieval  from  data  bases:  current  applications  to  maintenance  of  this 
topic  must  be  considered  only  on  the  fringes  of  AI,  with  the  possible  exception  of 
automatic  test  pattern  generation  (ATPG).  ATPG  is  used  to  generate  the  test 
patterns  for  digital  logic.  At  present,  this  machine  approach  is  superior  to  the 
human  in  formulating  tests  for  combinatorial  logic,  and  inferior  in  sequential  logic. 


Near  term  possibilities  are  the  improvement  of  machine  handling  of  sequential  logic 
and  development  of  ATPG  for  analog  circuitry.  The  ultimate  potential  would  be  the 
complete  elimination  of  the  manual  part  of  the  process." 

With  over  40  organizations  actually  pursuing  research  in  AI  use  with  database  systems 
(SIGART  Oct.  ’83),  we  see  this  to  be  a  very  active  topic.  A  few  products  are  even 
available  for  the  IBM-PC  (e.g.  R:Base,  TK  Solver).  The  goal  of  this  work  is  to  not  only 
speed  up  the  use  of  a  database,  but  to  find  more  relevant  information  for  the  problem  at 
hand.  Bibliographic  databases,  for  example,  typically  retrieve  less  than  40%  of  the 
relevant  material. 

The  systems  are  also  gaining  capability  to  retrieve  related  information  by  use  of  fuzzy  set 
theory,  special  database  machines,  and  by  use  of  a  special  IC  by  Proximity  Technology 
(Ft.  Lauderdale,  Florida)  which  can  accurately  find  information  even  with  misspellings  and 
interchange  of  characters.  Due  to  the  large  commercial  market,  progress  is  now  being 
made  quickly,  and  it  is  expected  that  some  results  in  intelligent  retrieval  will  be  usable  in 
testability  problem  solutions  within  the  next  five  years;  the  capability  of  starting  an 
application  within  three  years  appear  promising. 

2.4  EXPERT  CONSULTING  SYSTEMS 

Expert  systems  are  probably  the  hottest  topic  in  AI  today.  Expert  systems  have 
introduced  a  new  way  of  preserving  knowledge.  Such  a  system  is  able  to  retrieve 
and  process  stored  knowledge  to  perform  such  functions  as  diagnosis, monitoring, 
prediction, and  planning.  Currently, all  operational  expert  systems  are  rule  based; 
that  is, the  knowledge  is  stored  in  the  form  of  if-then  or  situation-action  rules. 
These  rules  (also  called  production  rules)  form  a  network  of  inferences  that  are 
used  to  perform  the  expert  functions. 

"It  has  become  fashionable  today  to  characterize  any  large,  complex  AI  system  that 
uses  large  bodies  of  domain  knowledge  as  an  expert  system.  Thus,  nearly  all  AI 
applications  to  real-world  problems  can  be  considered  in  this  category,  though  the 
designation  *know ledge-based  systems’  is  more  appropriate."  (6) 

An  expert  system  consists  of  (6): 

i)  a  knowledge  base  (or  knowledge  source)  of  domain  facts  and  heuristics 
associated  with  the  problem; 


2)  an  inference  procedure  (or  control  structure)  for  utilizing  the  knowledge  base 
in  the  solution  of  the  problem 

3)  a  working  memory— "global  data  base"— for  keeping  track  of  the  problem 
status,  the  input  data  for  the  particular  problem,  and  the  relevant  history  of 
what  has  thus  far  been  done 

Hundreds  of  articles  have  been  written  on  expert  systems,  and  over  20  of  the  most 
relevant  ones  are  in  the  bibliography. 

Teknowledge  (a  front  runner  of  dozens  of  knowledge  engineering  companies)  listed  some 
situations  that  need  knowledge  engineering;  often  in  the  form  of  expert  systems.  These 
situations  result  from  many  testability  problems: 

o  The  organization  requires  too  many  skilled  people  to  recruit  or  retain 
o  Problems  arise  that  require  too  many  possibilities  to  be  considered 
o  Job  excellence  requires  a  scope  of  knowledge  exceeding  reasonable  demands 
on  human  training  and  continuing  education 
o  Problem  solving  requires  several  people  because  no  single  person  has  the 
needed  expertise 

o  The  company's  inability  to  apply  its  existing  knowledge  effectively  now  causes 
management  to  work  around  basic  problems 

Teknowledge  also  listed  attributes  for  selecting  aif  appropriate  application,  many  of  which 
apply  to  testability  applications: 

o  Primarily  symbolic 

o  Commitment  and  availability  of  expert 

o  Importance  of  the  problem  (who  cares?) 
o  Scope  of  the  problem 

.  bounded  domain 

.  human  solves  problem  in  3  hrs.  to  three  weeks 
.  vocabulary  of  basic  concepts  consists  of  a  few  hundred  terms 
o  General  agreement  among  specialists  about  knowledge 


o  Data  and  test  cases  available 
o  Highly  combinatorial  problems  are  likely  candidates 
o  Incremental  progress  should  be  possible 

Knowledge  engineering  is  so  new  that  only  one  book  (by  Teknowledge)  has  been  written  on 
the  methods  of  building  expert  systems  (36).  With  dozens  of  ways  to  make  an  expert 
system  (table  2.4-1  from  (6)),  care  must  be  taken  in  selecting  the  right  tools  for  the  tasks. 
Far  too  often  this  survey  found  people  or  organizations  trying  to  use  an  AI  language  that 
happens  to  work  on  their  machine  to  make  an  expert  system  for  their  perceived  needs. 
Many  aerospace  contractors  have  realized  this  problem  and,  as  a  result,  formed  internal 
company  AI  centers  for  concentrating  knowledge  and  tools.  To  minimize  the  need  for 
knowledge  engineers,  IBM,  Boeing  and  others  are  developing  systems  that  are  expert  in 
eliciting  knowledge  about  a  domain  from  the  user.  This  is  one  of  several  techniques  that, 
when  available,  will  make  the  building  of  maintenance  expert  systems  more  cost 
effective. 


Maintenance  Expert  Systems 


The  maintenance  uses  of  expert  system  have  been  well  summarized  by  AIAM  (3): 


"Expert  systems  are  in  practical  use  today  for  configuring  computer  systems  (Rl)  and 
are  available  for  medical  diagnosis  (MYCIN,  CADUCEUS)  and  locating  mineral 
deposits  (PROSPECTOR).  There  are  also  expert  systems  in  various  states  of 
development  for  fault  isolation.  These  include  DELTA  for  the  maintenance  of 
locomotives,  REACTOR  for  nuclear  reactors,  CRITTER  for  digital  circuits,  IDT 
used  on  a  computer,  and  DART,  designed  for  computer  hardware.  (See  also  position 
paper  by  Caren  on  a  generic  maintenance  expert,  LES).  Hence,  maintenance  expert 
systems  are  relatively  near  term  possibilities  for  applying  to  testability. 

The  diagnostic  search  programs  (LOGMOD,  STAMP,  FIND),  mentioned  above,  would 
seem  to  be  a  valuable  adjunct  to  an  expert  maintenance  program  in  that  they  can  be 
used  by  the  program  to  set  up  its  strategy  from  the  design  of  the  unit  under  test. 

An  expert  system  requires  a  knowledge  base  of  its  domain.  At  present,  this  is 
provided  by  the  user.  LOGMOD,  et  al,  require  user  layout  of  the  system  and  user 
generation  of  appropriate  tests.  The  ultimate  expert  system  would  use  advances  in 
automatic  programming  to  perform  fully  automatic  testing  by  deriving  its  know¬ 
ledge  from  a  machine-read  schematic,  forming  its  strategy  from  that  knowledge, 
and  generating  appropriate  tests  as  it  needs  them. 


TABLE  2.4-1  Control  Structures  of  Some  Well  Known  Expert  Systems 


Once  the  ATE  has  the  knowledge  it  needs  to  operate  as  an  expert  system,  it  can  also 
be  used  as  a  maintenance  aid  to  direct  a  human  when  his  intervention  is  necessary. 
It  could  also  provide  the  interaction  needed  lor  training  its  operator.  Thus,  the 
expert  system  would  serve  as  ATE,  maintenance  aid,  and  training  device.  While 
probably  not  a  short  term  effort,  the  integration  of  maintenance  aids  should  not  be  a 
far-out  proposition.  Integrating  a  training  capability  will  be  a  long  term  proposition 
needing  advances  in  computer  aided  instruction  (CAI)  techniques. 

Computer  aided  design  (CAD)  programs  can  be  considered  a  form  of  expert  system, 
even  though  their  current  approaches  may  not  be  considered  to  be  examples  of  AI. 
The  Rome  Air  Development  Center  is  studying  the  incorporation  of  ease-of-test 
considerations  to  CAD  programs.  From  these  can  spring  an  expert  system  which 
will  create  a  design  with  a  minimum  of  maintenance  problems.  This  will  require  the 
creation  of  a  set  of  design  rules  for  ease  of  maintenance  and  a  set  of  rules  to  trade 
these  off  against  other  considerations  such  as  thermal  design,  wiring  constraints, 
etc.  The  design  rules  should  be  a  near  term  possibility,  but  creating  the  trade-off 
rules  will  be  a  more  difficult  task." 

The  many  maintenance  expert  systems  surveyed  were  created  by  over  a  dozen  different 
methods,  only  a  few  of  which  are  generic  and  can  be  adapted  to  another  system  with  little 
or  no  change  in  its  rule  structure  (8,  9,  10,  11).  However,  Pau  (11)  has  developed  over  400 
diagnostic  metarules  which  appear  to  form  an  excellent  structure  for  many  maintenance 
expert  systems.  Dr.  Pau  uses  a  nested  set  of  decision  tables  similar  to  failure  modes  and 
effects  analysis  (FMEA)  to  store  the  information  (figure  2.4-1).  The  resulting  diagnostic 
system  architecture  includes  a  learning  database  (figure  2.4-2). 


Another  approach  to  reducing  the  cost  to  generate  and  validate  an  expert  system  is  to 
have  it  generate  its  own  knowledge  base  from  observations.  Mitchie  suggested  that  "most 
experts  learn  by  example,  so  why  shouldn't  an  expert  system?".  The  machine  learning 
approach  is  a  breakthrough  technique.  It  is  discussed  in  section  2.1 1. 


The  third  major  improvement  in  effectiveness  of  a  maintenance  expert  would  come  from 
expanding  the  current  shallow  knowledge  of  a  unit  under  test  to  include  deep  knowledge. 
Kinnucan  (28)  provides  a  good  survey  of  research  in  deep  electronic  knowledge: 


Figure  2.4-1  Decision  Table 


£  end  alternatives 


Figure  2.4-2  Diagnostic  System  Architecture 
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"A  group  headed  by  Randy  Davis  at  MIT  is  developing  an  expert  system  for  diagnosing 
electronic  equipment,  a  system  based  on  knowledge  of  the  structure,  function,  and 
behavior  of  circuits-  This  system  will  represent  both  the  physical  and  functional 
structure  of  the  syst<.  as  defined  by  a  schematic. 

These  systems  work  by  pathways  of  causation.  Representation  of  both  physical  and 
functional  structure  of  the  system  is  necessary  because  functionally  adjacent 
components  may  be  physically  separated.  For  example,  the  circuits  that  make  up  a 
single  subsystem  may  be  physically  separated  on  a  printed  circuit  board.  Yet  two 
physically  adjacent  circuits  may  belong  to  quite  different  functional  units.  Physical 
representation  allows  the  system  to  detect  faults  such  as  a  solder  bridge  between 
two  pins  of  an  integrated  circuit,  as  well  as  those  that  can  be  logically  traced 
through  functional  linkages. 

The  system  models  devices  by  treating  them  as  black  boxes  with  input  and  output 
ports  that  are  in  turn  connected  to  other  devices.  The  normal  function  of  the  device 
is  represented  by  simulation  rules  that  specify  an  output  for  a  given  input,  and 
inference  rules  that  state  what  can  be  inferred  about  the  inputs  of  the  device  from 
the  values  of  its  outputs,  assuming  the  device  is  functioning  normally.  This 
representation  is  hierarchical.  Each  of  the  major  components  of  a  system  can  be 
broken  down  into  smaller  parts.  This  allows  the  expert  system  to  deal  with  the 
enormous  complexity  of  computers  and  other  electronics  systems  (although  a 
complete  model  of  even  a  modest  size  computer  is  not  going  to  be  predicted  in  the 
near  future). 

It  makes  diagnoses  by  a  procedure  Davis  calls  'discrepancy  detection'.  The  system 
simulates  the  behavior  of  a  suspect  device  to  establish  its  normal  outputs.  It  then 
compares  the  device's  actual  output  with  the  simulated  output.  If  there  is  a 
discrepancy,  the  component  is  known  to  be  faulty." 

In  interviews  with  both  Davis  (27)  and  Generseth  (28),  they  concluded  that  this  work  is  not 
expected  to  be  useful  for  field  applications  for  at  least  5  years.  When  it  does  become 
practical,  an  electronic  maintenance  expert  system  will  need  on-site  real-time  computa¬ 
tion  capability,  which  implies  use  of  at  least  the  capability  of  an  engineering  workstation 
or  special  purpose  AI  machine  (sec  5.11). 


2.5  ROBOTICS 


At  this  time  robotics  is  being  used  in  testability  application  research  to  place  boards  on 
testers  and  for  guided  probe  placement.  It  is  being  enhanced  by  vision  and  other  sensory 
applications  such  as  the  ability  to  "feel"  pressure.  This  application  is  so  new  that  bugs  and 
reliability  problems  still  exist  in  demonstration  units. 


Use  of  robotics  is  not  economically  feasible  for  factory,  organizational  level  and  most 
depot  level  military  testability  applications  for  at  least  five  years,  as  the  current  state  of 
robotics  requires  very  predictable,  high  volume  operation  for  economic  use. 

2.6  AUTOMATIC  PROGRAMMING 

Our  survey  shows  that  automatic  programming  has  been  used  to  analyze  the  structure  of  a 
program  and  in  determining  how  the  resulting  analysis  can  be  used  to  guide  a  proof  of 
correctness  for  that  program.  More  recent  research  shows  that  automatic  programming 
approaches  have  been  used  for  program  synthesis.  With  the  results  from  this  research,  the 
discipline  may  have  an  application  to  testability.  However,  because  of  the  scarcity  of 
information  in  this  area  and  the  lack  of  specific  research  applicable  to  testability, 
automatic  programming  is  judged  to  be  more  than  five  years  away. 

"ATPG  can  be  considered  an  example  of  automatic  programming.  There  are  also 
programs  which  assist  a  test  engineer  in  creating  ATLAS  statements  for  test 
equipment.  Though  these  are  not  really  sophisticated  enough  to  warrant  the  title. 
The  ultimate  application  of  automatic  programming  will  be  the  real  time  generation 
of  ATLAS  test  vectors  from  a  computerized  representation  of  the  system,  as  they 
are  needed  by  an  expert  system  isolating  a  failure.  This  is  definitely  not  a  short 
term  goal."  (3) 

Much  work  in  automatic  programming  is  needed  for  it  to  be  applicable  to  the  highly 
controlled  software  environment  of  the  military. 

2.7  PERCEPTION  PROBLEMS 

The  most  advanced  use  of  perception  is  shown  in  several  of  Dr.  Pau's  papers  (11,  12,  13, 
14,  15).  In  "Integrated  Testing  and  Algorithms  for  Visual  Inspection  of  Integrated 
Circuits"  (12),  he  combines  electrical  testing  with  visual  image  analysis  before  the  lid  is 
put  on  the  IC.  This  test  expert  system  could  be  extended  to  PC  board  inspection  by 
computer  using  visible  or  infra-red  light  combined  with  electrical  testing  for  fault 
diagnosis. 

In  Pau  (14)  "Application  of  Pattern  Recognition  to  Failure  Detection  and  Analyses  in  Non- 
Digital  Systems,"  perception  techniques  are  discussed  that  could  be  applied  to  detect 


degradation  of  mechanical  parts  (engine,  etc.)  before  they  are  destroyed  by  bearing 
failure,  excessive  vibration,  etc. 

CR  Technology  (1701  Reynolds,  Irvine,  California)  has  a  pattern  recognition  system  that 
connects  to  ATE  to  read  6  lines  of  characters  with  a  television  camera,  listen  to  tones  and 
manipulate  switches.  This  system  is  expected  to  have  complete  (or  extended)  pattern 
recognition  capability  to  "read”  the  fully  electronic  dials  of  new  aircraft.  This  is  the  first 
example  of  a  product  with  embedded  AI  that  can  be  used  in  the  situation  test.  But  while 
it  eliminates  the  technician's  need  to  confirm  displays,  it  does  not  produce  the  large  scale 
cost  savings  desired  by  this  study. 

However,  an  additional  driver  is  that  in  order  to  discriminate  false  alarms  from 
intermittents,  pattern  perception  techniques  must  be  developed  to  process  the  data  that 
will  be  available  in  3  to  5  years  from  Smart  BIT. 

2.8  COMBINATORIAL  AND  SCHEDULING  PROBLEMS 

Combinational  techniques  can  be  used  in  finding  the  optimal  path  in  a  network.  In 
testability  these  could  theoretically  be  used  in  obtaining  the  least  time  or  least  cost  path 
to  finding  faults  using  a  large  ATE  program.  No  references  to  application  of  these 
techniques  with  any  relevance  to  testability  were  found. 

Scheduling  problems  have  historically  been  solved  best  using  operations  research  tech¬ 
niques;  they  can  also  be  solved  through  use  of  AI.  Scheduling  techniques  could  be  used  in 
the  near  term  to  develop  models  to  analyze  maintenance  queueing  for  minimum  impact  on 
system  down  time.  But  with  the  assortment  of  mature  non-AI  techniques  now  used  to 
accomplish  these  types  of  tasks,  AI  scheduling  is  probably  not  of  economic  interest  for 
testability. 

2.9  VOICE  SYNTHESIS  AND  RECOGNITION 

Voice  synthesis  is  now  a  tool  being  applied  to  commercial  systems  for  restricted 
vocabulary  voice  output  from  computer.  Voice  recognition,  even  with  heavy  RicD 


funding,  is  not  yet  an  easily  applied  tool.  The  Japanese  are  expected  to  be  front  runners 
in  voice  recognition  with  their  5th  generation  computing  project  and  only  500  syllables  in 
their  language  that  need  to  be  recognized. 

Voice  Response  (synthesis) 

On  some  ATE  (Fairchild  series  70),  voice  synthesis  allows  the  operator  to  keep  his  eyes  on 
the  unit  under  test  while  performing  diagnostics.  The  Texas  Instruments  Professional 
Computer  has  one  of  the  best  synthesis  systems  for  personal  computers.  There  are  many 
synthesis  techniques  used  to  electronically  generate  the  40  English  phonemes. 

Voice  Recognition 

With  10,000  syllables  and  100,000  English  words  that  need  to  be  recognized,  automatic 
dictation  is  not  expected  until  the  1990's.  "Widespread  use  of  voice  recognizers  await  the 
availability  of  low  cost  connected  speech  systems  achieving  better  than  99%  accuracy 
with  limited  vocabularies— 100  words."  (6)  There  are  indications  that  a  speech  under¬ 
standing  system  using  a  Natural  Language  parser  will  be  introduced  by  IBM  in  the  mid 
80's.  (6).  CAD/CAM  systems  are  also  now  becoming  available  with  voice  input  for  the 
control  applications,  leaving  the  hands  free  to  use  a  graphics  tablet,  light  pen  or  mouse. 

In  the  future,  voice  input  might  be  a  more  efficient  way  for  a  user  to  provide  information 
to  the  ATE;  however,  the  current  graphics  menu  input  system,  with  mouse,  track  ball, 
etc.,  may  be  as  fast  and  is  a  proven  low  risk  technology.  While  data  compression 
techniques  have  allowed  voice  synthesis  to  be  done  on  special  IC's,  even  100  word 
recognition  still  requires  a  fast,  powerful  computer,  and  training  is  required  for  a  system 
to  understand  a  speaker's  natural  voice  variations  and  moods.  The  Personal  Electronic  Aid 
for  Maintenance  project  at  TI  (25)  will  have  some  speech  recognition  capability  and  its 
progress  is  worth  following.  Speech  recognition  is  a  high  risk,  high  cost  AI  technique  for 
the  next  three  to  five  years. 


2.10  INTELLIGENT  COMPUTER  AIDED  INSTRUCTION 


AI  techniques  are  being  used  to  improve  the  quality  of  computer  aided  instruction  (CAI). 
The  AI  version  of  CAI  is  usually  called  Intelligent  Computer  Aided  Instruction  (ICAI).  It 
differs  from  traditional  CAI  in  that  the  deterministic  programming  of  the  order  of 
presentation  of  material  is  replaced  with  AI  techniques.  Thus  the  system  has  the 
capability  to  better  adapt  to  the  student's  individual  needs;  whole  sections  of  instructions 
may  be  skipped  (or  emphasized)  by  the  AI  program's  assessment  of  the  student's  progress. 

ICAI  systems  consist  of:  (1)  problem-solving  expertise,  which  is  the  knowledge  the  system 
tries  to  impart  to  the  student;  (2)  the  student  model,  which  monitors  what  the  student 
does  and  does  not  know;  and  (3)  tutoring  strategies,  which  specify  how  the  system  presents 
material  to  the  student. 

Our  investigation  in  this  area  shows  that  the  application  of  AI  techniques  to  CAI  will  not 
have  a  significant  effect  on  testability  quality  or  provide  a  cost  savings  in  the  near  term. 

In  fact,  current  research  has  failed  to  show  any  significant  improvement  on  the  learning 
curve  of  the  student  by  use  of  CAI  or  ICAI.  The  use  of  video  disks  may  give  CAI  a  wider 
range  of  teaching  techniques,  but  the  techniques  used  by  a  good  teacher  have  yet  to  be 
encoded  into  a  computer  (40).  A  new  journal,  Machine- Mediated  Learning,  contains  many 
items  on  ICAI.  Intelligent  Tutoring  Systems  441)  is  a  good  overview  book.  Further 
references  are  found  in  the  maintenance  training  part  of  section  8.2. 

Richardson  (5)  reviews  12  Intelligent  CAI  systems  and  concludes  that,  while  no  robust 
system  now  exists  for  military  use,  more  effort  in  the  area  will  produce  usable  results. 
The  time  frame  for  practical  application  in  the  military  is  at  least  3  years  off,  probably 
5  years  or  more. 

2.11  MACHINE  LEARNING  (Decision  Tree  Induction) 

Machine  learning  is  an  active  area  in  artificial  intelligence  which  combines  several  AI 
techniques:  expert  systems,  intelligent  retrieval  from  databases,  and  pattern  recognition. 
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Machine  learning  systems  have  been  developed  and  applied  to  a  wide  variety  of  diagnostic 
problems  (17,  19,  20).  Machine  learning  is  defined  as  constructing  an  organized 
representation  of  experience  by  a  computer.  The  possible  types  of  machine  learning 
include: 

o  learning  by  being  constructed  or  programmed  (rote) 
o  learning  from  instruction  (by  being  told) 
o  learning  by  deduction 

o  learning  by  analogy 

o  learning  from  examples 

o  learning  form  observation  and  discovery 

It  is  widely  recognized  that  automatic  learning  using  artificial  intelligence  techniques 
within  an  unstructured  domain  is  a  very  difficult  task  and  that  little  in  the  way  of 
practical  application  has  been  demonstrated.  However,  when  the  task  domain  is 
restricted  to  that  of  learning  from  observation  of  data  presented  in  tabular  form,  it  has 
been  shown  to  be  a  practical  technique.  To  be  usable,  the  table  must  have  all  relevant 
attributes  for  the  diagnostic  decisions.  The  expert  does  not  have  to  tell  the  relevance  of 
each  attribute,  but  every  attribute  which  might  have  some  relevance,  even  if  it  is  not 
used  consciously  in  his  diagnostic  decision,  must  be  included.  As  is  true  with  human 
learning  from  observation,  both  positive  and  negative  examples  of  diagnostic  decisions 
must  be  included  in  the  tables.  Testability  applications  of  machine  learning  could  use 
observation  of  the  diagnostics  performed  by  an  expert  to  infer  the  rules  that  he  used. 

Machine  learning  appears  to  have  the  most  potential  for  the  military  testability  situation 
by  being  applied  to  fault  diagnosis.  The  following  are  some  of  the  attributes  that  would 
have  to  be  in  a  data  base  in  tabular  form  for  machine  learning  to  perform  fault  diagnoses 
for  electronic  equipment: 
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Field  fault  information 
Tester  data  (not  just  GO/NOGO  results) 
Test  results  from  other  relevant  test  steps 
Functional  signal  dependency  table 
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Results  of  repair  (did  the  repair  cure  the  problem) 
o  Fault  history  of  unit 

o  Diagnostic  decisions  by  technicians 

o  Intermittent  failures  (SMART  BIT  generally  needed  to  obtain  this  fault  data) 

Dr.  Michalski's  work  at  the  University  of  Illinois  has  produced  a  machine  learning  system 
which  forms  a  knowledge  engineer's  workbench  for  tabular  data.  His  Intelligence 
Systems  Company  of  Urbana,  Illinois,  is  marketing  an  "Assistant  for  Building  Better 
Expert  Systems"  (ABBESS),  which  determines  attribute  weighting  by  use  of  a  variable 
valued  logic  system.  Each  attribute  should  have  less  than  20  values  for  efficient  learning 
with  ABBESS;  this  is  adequate  to  describe  the  attributes  for  electronic  diagnosis. 
Instead  of  merely  recording  if  a  test  step  passed  or  failed,  the  following  nine  catagories 
of  test  result  could  be  recorded: 

o  Passed  test 

o  Passed  test  but  noisy 

o  Passed  test  but  oscillation  present 

o  Passed  test  low  (or  high) 

o  Failed  test  but  almost  passed  (low  or  high) 
o  Failed  test  low  -  no  signal 
o  Failed  test  low  -  noise 
o  Failed  test  high  -  distorted  signal 
o  Passed  test  -  signal  distortion 

The  diagnostic  and  test  attributes  described  above  provide  the  learning  program  with 
sufficient  information  to  automatically  determine  the  significant  attributes  of  the  man- 
generated  diagnostics. 

Michalski  (18)  compared  machine  learning  using  a  program  similar  to  ABBESS  with  18 
other  numerical  taxonomies  for  two  classification  applications  —  selection  of  micro¬ 
computers  and  diagnosing  soybean  diseases.  Machine  learning  appears  to  have  capability 
for  building  expert  systems  in  less  than  1/10  the  time  required  for  standard  rule-based 
techniques,  while  generating  more  precise  production  systems  than  those  obtained  from 
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experts.  This  suggests  that  machine  learning  by  observation  could  provide  a  better  tool 
for  diagnostic  analysis  than  those  previously  used. 

Michalski*s  publication  on  diagnosis  of  soybean  disease  provides  us  with  a  concise 
example  of  machine  learning  of  the  diagnosis  process  used  by  humans.  The  soybean 
disease  diagnosis  involves  the  use  of  training  data  from  a  soybean  expert.  The  structure 
of  the  soybean  application  applies  to  electronic  diagnosis  and  improvement  of  Automatic 
Test  Program  Generation,  even  if  the  content  of  the  tabular  data  does  not.  The  soybean 
expert  was  trying  to  classify  15  diseases  by  analysis  of  35  multivalued  attributes,  some 
of  which  were: 

o  Time,  rain,  temperature,  hail 
o  Number  of  years  crop  repeated  in  same  location 
o  Damaged  area  of  plant,  plant  height 

o  Leaf  spots  and  spot  size,  mildew  or  seed  mold  and  seed  size 

The  machine  learning  system  could  correctly  diagnose  a  disease  after  using  only  290 
training  examples.  This  was  compared  to  another  expert  system  using  rules  extracted 
from  a  botany  expert.  Comparison  between  the  two  systems  follows: 


First  Choice 

One  of 

Two  Choices 

Not 

Type 

Correct 

Correct 

Diagnosed 

Machine  learning 

98% 

100% 

0% 

Expert  system 

72% 

97% 

2% 

Both  the  machine  learning  and  the  conventional  expert  system  were  forced  to  limit 
diagnosis  to  2  choices,  and  order  them  by  priority.  The  rule  based  expert  system  had  the 
correct  diagnosis  as  the  first  choice  72%  of  the  time  and  had  the  correct  diagnosis  as  one 
of  the  two  choices  97%  of  the  time.  It  failed  to  make  any  diagnosis  in  2%  of  the  cases. 
ABBESS  produced  much  better  results,  always  making  a  diagnosis.  In  more  complex 
learning  of  diagnoses  ABBESS  has  used  up  to  100,000  examples  to  develop  crisp  rules. 


This  very  large  data  base  is  far  beyond  human  capacity  to  fully  understand  or  translate 
into  an  expert  system. 


ABBESS  illustrates  that  it  is  not  essential  to  use  a  regular  AI  language  (e.g.,  LISP)  or 
machine  (e.g.,  Symbolics,  Dolphin,  LMI).  The  ABBESS  system  is  implemented  on  a  DEC 
VAX  computer  using  30,000  lines  of  Pascal  code.  This  Pascal  code  is  able  to  perform 
many  of  the  functions  of  LISP,  the  preferred  artificial  intelligence  language.  The 
resulting  production  rules  (diagnostic  decision  tree)  can  be  run  on  a  much  smaller  Pascal 
based  machine.  Currently,  an  IBM  PC  is  used  for  rule  execution  after  learning  is 
complete. 

Another  machine  learning  program,  from  England,  has  less  capability  than  A-.^ESS; 
however,  it  runs  entirely  on  an  IBM  PC  using  a  spread  sheet  type  of  data  entry.  Expert 
Ease  was  briefly  reviewed  (Datalink,  June  27,  1983)  and  is  available  from  Export 
Software  International  4  Canongate  Venture.  New  Street,  Royal  Mile,  Edinburough, 
Scotland  EH8,  8BH. 

2.12  CONCLUSIONS 

Computer  systems  now  have  sufficient  computational  capacity  that  can  be  used  to 
understand  and  help  the  user,  rather  than  just  obey  some  precisely  formulated 
commands.  Artificial  intelligence  provides  a  set  of  techniques  for  allowing  the  person 
and  computer  to  work  as  a  team,  with  each  member  of  the  team  having  special 
capabilities  (figure  2.12-1)  (31). 

Expert  systems  is  the  most  active  development  in  artificial  intelligence,  and  is  now 
capable  of  being  used  with  several  testability  applications  as  a  knowledge  manager.  An 
expert  system  could  be  used  to  : 

o  Get  knowledge  into  a  computer 

o  Test  it 

o  Debug  it 
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FIGURE  2.12-1  Man  and  AI  Can  Work  Together  Harmoniously 


o  Extend  it 

o  Modify  it,  and  finally 

o  Give  the  knowledge  back  to  the  human  in  improved  form 

Diagnostic  metarules  and  machine  learning  will  allow  maintenance  expert  system  tools 
to  be  developed  which  can  be  easily  applied  to  many  projects.  Presently  most  expert 
systems  are  developed  through  the  knowledge  engineering  lifecycle  (figure  2.12-2)  for 
each  project  with  the  knowledge  coming  from  a  single  human  expert.  Maintenance 
expert  system  tools  have  been  created  (8,  9),  including  some  which  can  incorporate 
diagnostic  experience  (10,  1 1).  The  use  of  system  independent  diagnostic  metarules  will 
permit  a  maintenance  expert  system  to  be  created  without  the  extensive  refining, 
redesigning,  reformatting  feedback  shown  in  figure  2.12-2.  These  techniques  should 
reduce  the  time  to  create  an  expert  system  for  maintenance  from  five  manyears  to  one 
manyear  for  commercial  systems.  By  using  extensive  validation  and  verification  of 
metarules,  allowing  the  users  to  modify  the  data  only,  and  by  having  internal  rules  for 
data  consistency  checking,  maintenance  expert  system  tools  should  be  able  to  be 
developed  for  military  applications. 

During  the  next  three  years  no  new  A1  developments  in  understanding  circuit  functions 
and  faults  through  "deep  knowledge"  is  expected.  It  may,  however,  be  possible  that  AI 
can  be  teamed  with  a  system  that  already  understands  circuit 
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Figure  2.12-2  Knowledge  Engineering  Lifecycle 


functions  (i.e.,  the  engineering  work  station).  This  combination  would  be  able  to 
simulate  circuit  operation  and  faults  while  a  unit  is  being  diagnosed,  without  requiring 
the  ten  to  fifty  manyears  of  development  if  done  by  AI  alone. 

The  summary  of  the  status  of  AI  techniques  for  application  to  military  testability  is 
shown  in  table  2.12-1.  Although  some  techniques  are  useful  to  military  testability,  most 
would  not  yield  significant  cost  benefits  in  the  near  term,  if  ever.  This  topic  in  the  form 
of  life  cycle  cost  benefits  of  potential  AI  applications,  will  be  discussed  later  in  the 
study. 


Tool  can 

Substantial 

be  used 

Research 

progress 

now 

activity 

in  2-5  years 

Useful 

Required 

Theorem  Proving 

Natural  Language 

* 

** 

* 

* 

Processing 

Intelligent  Retrieval 

* 

#* 

* 

* 

* 

from  Data  Base 

Expert  Consulting  System 

** 

** 

* 

** 

* 

Robotics 

* 

Automatic  Programming 

* 

* 

* 

* 

Perception 

* 

Combination  and 

# 

** 

Scheduling  Problems 

Voice  Recognition 

* 

* 

* 

Voice  Synthesis 

* 

* 

Intelligent  Computer 

# 

«* 

Aided  Instruction 

Machine  Learning 

* 

# 

• 

* 

Table  2.12-1.  AI  Technique  Status  for  Military  Testability  Applications 
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3.0  SURVEY  OF  EXISTING  AND  DEVELOPING  BIT  AND  ATE 


The  two  primary  methods  of  testing  military  electronics,  Built-In  Test  (BIT)  and 
Automatic  Test  Equipment  (ATE),  have  reached  limits  of  capability  that  still  require 
excessive  manpower  for  maintenance.  A  survey  was  made  of  BIT  and  ATE  problems  and 
the  potential  for  near  term  application  of  artificial  intelligence.  Information  was 
obtained  from  published  literature  and  reports,  interviews  with  ATE  vendors,  and  relevant 
R&D  groups  in  government,  industry  and  universities. 

The  key  BIT/ ATE  findings  show  a  lack  of  uniform  application  of  Design  for  Testability 
(DFT)  techniques: 

o  Present  application  of  AI  to  BIT  and  ATE  is  practically  nonexistent 
o  Designers  not  given  incentive  or  feedback  on  DFT  for  BIT/ATE 
o  Inadequate  management  perception  of  DFT  quality 
o  ATE  problems  generally  due  to  lack  of  adequate  DFT  and  BIT 
o  ATE  test  problems  increased  because  BIT  data  is  often  ignored 
o  BIT  experts  needed  since  BIT  is  frequently  designed  new  for  each  military 
acquisition 

Military  BIT  and  ATE,  as  applied  now,  are  expensive  to  implement.  Neither  one  is 
specifically  supported  by  automated  design  tools,  although  general  CAD  tools  are 
applicable.  This  survey  revealed  opportunities  for  artificial  intelligence  to  provide 
improvements  to  both  BIT  and  ATE,  especially  where  it  supported  DFT  decisions  made  in 
the  early  phases  of  a  project  concerning  tradeoffs  between  BIT  and  ATE. 

The  designer  frequently  is  not  aware  of  the  burden  that  a  lack  of  BIT  (or  insufficient  BIT) 
or  other  DFT  shortcomings  places  on  the  ATE  or  the  maintenance  technician.  Since  the 
designer  frequently  has  no  means  to  measure  how  well  BIT  or  ATE  will  do  his  job,  there  is 
no  feedback  to  gain  the  expertise  that  improves  designs. 

There  has  been  a  tendency  to  ignore  testing  of  complex  electronics  as  a  design  factor 
during  early  phases  of  a  project.  This  results  in  much  more  complexity  in  the  ATE  which, 
in  turn,  becomes  very  costly  because  it  must  bear  ail  of  the  test  impact.  By  inserting 
better  testability  through  BIT  into  the  early  design  phases,  the  portion  of  life  cycle  costs 
due  to  testing  can  be  reduced. 


3.1  BUILT-IN  TEST  (BIT) 


The  use  of  BIT  is  increasing  but  the  application  results  have  been  poor  due  to  nonuniform 
methods  and  lack  of  verification  techniques.  As  a  result,  operators  and  technicians  have 
no  faith  in  BIT  with  its  high  rate  of  alarms  which  cannot  be  traced  to  a  cause. 

The  BIT  considerations  in  Table  3.1-1  (31)  show  that  either  a  BIT  design  expert  is  needed 
to  make  the  right  BIT  the  first  time,  or  else  the  design  must  evolve.  Unfortunately,  a 
program's  time  constraints  frequently  do  not  permit  much  design  evolution. 

3.1.1  BIT  Trends 

Military  BIT  experiences  excessive  false  alarm  rates,  caused  by  incorrect  BIT  design  and 
intermittent  and  transient  faults  due  to  effects  which  were  not  (or  could  not)  be 
considered  during  design.  There  is  work  in  progress  to  reduce  the  rate  of  false  alarms  by: 
o  Developing  new  false  alarm  filters  (Malcolm  46) 

o  Providing  better  built-in  test  trades  at  the  early  phase  of  design,  and  providing 
for  testability  data  collection,  analysis  and  corrective  action,  (MIL  STD 
XXXX,  (21)  RADC  Testability  Notebook  (22)) 
o  Recording  field  built-in  test  results  for  maintenance  use  (1) 
o  Developing  a  SMART  BIT  to  incorporate  false  alarm  filters,  fault  recording, 
etc.  (RADC  Contract,  F30602-84-C-00 51,  in  work) 

This  survey  found  no  work  in  progress  on  simulating  intermittents  and  other  false  alarm 
conditions  during  the  design  phase. 

The  following  factors  all  tend  to  reduce  the  amount  of  BIT  implemented  in  military 
designs,  thus  pushing  the  burden  of  test  on  the  ATE: 
o  Lack  of  expert  BIT  designers 

o  High  cost  of  reinventing  BIT  for  each  program  (no  BIT  standardization) 
o  Lack  of  management  visibility  tools  to  help  judge  quality  of  BIT  design 

o  Support  organizations  have  minimal  involvement  in  weapons  procurement 

Commercial  incorporation  of  BIT  designs  for  computers,  self-checking  integrated  circuits, 
and  VLSI  self  test  have  proven  that  it  is  cost  effective  to  incorporate  BIT  capability  when 
a  complex  product  is  being  marketed  for  high  volume  applications.  The  opportunity  now 


Table  3. 1-1.  Generic  Issues  in  Designing  Bit  (31) 


exists  to  develop  standardized  BIT  approaches  which  can  be  used  in  many  military 
applications  and  standard  verification  techniques  to  validate  them  (RADC  Contract:  BIT 
Verification  Techniques  F30602-84-C-0021  in  work). 

3.1.2  AI  Applications  to  BIT 

Development  of  standard  BIT  functions  that  could  be  implemented  in  a  general  purpose 
BIT  module  would  be  of  significant  benefit  to  the  military.  A  weapons  program  should  not 
expect  to  create  a  generic  BIT  within  its  program  schedule,  just  as  it  does  not  expect  to 
develop  a  new  microprocessor  for  a  program.  To  be  practical,  the  BIT  must  easily  adapt 
to  applications,  conditions,  and  the  users'  requirements.  It  must  also  capture  information 
from  hard  and  intermittent  faults  which  can  be  used  as  an  aid  in  subsequent  diagnosis. 
Such  a  BIT  design,  which  has  had  extensive  verification  and  validation  for  the  military 
environment,  would  permit  a  low  cost,  high  quality  BIT  to  be  used  on  many  programs. 

The  same  forces  which  have  resulted  im  highly  sophisticated,  high  density  electronics  now 
permit  creation  of  a  highly  sophisticated,  high  density  BIT.  Integrated  circuit  technology 
provides  the  opportunity  to  design  a  BIT  element  which  can  adapt  to  many  future  program 
needs.  Technology  trends  toward  denser  circuit  packaging  and  larger  die  sizes  also 
support  this  approach. 

Initially,  the  development  of  a  BIT  chip  set  (or  IC,  in  its  2nd  generation),  which 
incorporates  good  false  alarm  filtering  and  has  been  extensively  verified  and  validated  for 
the  military  environment  would  provide  a  very  low  cost  BIT  solution  for  many  programs. 
A  BIT  chip  set  would  require  minimal  board  area  while  allowing  a  proven  design  to  be 
incorporated  at  low  risk.  Project  offices  generally  require  new  programs  to  have  BIT 
detection  and  isolation  in  excess  of  95%,  with  a  false  alarm  rate  of  less  than  5%.  These 
goals  are  difficult  to  achieve  if  BIT  is  designed  concurrently  with  the  rest  of  the  program. 
An  available  standard  BIT  IC  or  chip  set  would  make  it  easier  to  meet  those  requirements. 
This  chip  set  should  permit  not  only  data  recording  and  prog  ammability  but  also  support 
use  of  AI  features.  The  RADC  Smart  BIT  development  effort  which  will  attempt  to  use 
AI  techniques  may  lead  to  such  a  capability  (See  section  5.4  for  a  description). 


3.2  ATE  SURVEY 


The  results  of  the  ATE  portion  of  the  survey  are  summarized  in  figure  3.2-1,  which 
relates  BIT  and  ATE  problem  causes  to  their  impact  on  test  creation,  test  execution,  and 
test  maintenance.  This  section  describes  the  problem  areas  and  their  causes,  and  then 
discuss  the  merits  of  curing  &  cause  versus  providing  symptom  relief.  The  conclusion  of 
this  section  describes  general  solutions  that  will  be  discussed  in  more  detail  and  applied  to 
specific  applications  in  Section  5.0. 

3.2.1  ATE  Problem  Areas 

Problems  in  ATE  affect  cost  and  capability  in  the  creation,  use,  and  maintenance  of  the 
test  program  set  (TPS)  which  includes  both  the  test  software  and  the  interface  adapter. 
Kunert  (42)  provides  a  brief  review  of  "ATE  Applications  of  Artificial  Intelligence"  with 
suggested  use  of  expert  systems  for: 
o  System  fault  isolation 

o  CAD  with  testability  constraints 

o  Fault  diagnosis  of  a  Unit  Under  Test  (UUT)  at  the  depot 

3.2.1. 1  Test  Program  Set  Creation  Problems 

Test  program  cost  and  capability  are  a  function  of  testability.  As  the  number  of  gates  to 
be  tested  increases,  the  relative  cost  for  test  generation  increases  exponentially  when 
design  for  testability  is  ignored  (See  figure  3.2-2).  Figure  3.2-3  shows  that  the  number  of 
tests  needed  to  achieve  a  given  test  coverage  level  is  also  a  strong  function  of  testability. 
Devices  may  not  be  fully  tested  due  to  cost  or  creation  time  limitations.  Test  program 
sets  are  also  very  expensive  to  create  when  automatic  test  program  generation  or  other 
tools  are  not  available,  as  is  frequently  the  case  for  the  military  ATE. 
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Figure  3.2-2  Cost  Advantage  of  Design  for  Testability 
(Muchldorf  IEEE  TC  3an.  1980) 


3.2. 1.2  Test  Program  Set  Usage  Problems 

Problems  in  using  test  programs  on  ATE  include  slow,  sequential  execution  of  end-to-end 
test,  retest  OK,  and  diagnostic  knowledge  which  is  slowly  acquired  by  performing  tests  for 
many  months  then  lost  when  diagnostic  expert  has  left  (tour  of  duty  ended). 
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3.2.1. 3  Test  Program  Set  Maintenance  Problems 

The  TPS  must  be  maintained  to  incorporate  changes  in  units,  support  software  (such  as 
ATLAS  compiler),  ATE  configuration,  and  diagnostics.  General  improvements  in  TPS 
coverage  or  diagnostic  capability  are  frequently  not  incorporated;  due  to  the  extreme  cost 
of  military  changes,  there  is  a  tendency  to  incorporate  only  those  change  requests  which 
stop  good  units  from  failing  tests. 

3.2.2  Causes  of  ATE  Problems 

The  major  causes  of  ATE  problems  are  the  lack  of  design  for  testability  features  in  the 
units  being  tested  and  the  inadequate  ability  of  management  to  measure  DFT  quality. 

The  lack  of  ATE  speed  or  number  of  steps  that  can  be  executed  without  stopping  (program 
depth)  are  less  important  problem  causes  and  are  also  a  secondary  effect  of  the  lack  of 
DFT  features  in  the  unit  under  test.  There  is  currently  no  incentive  for  the  contractor  to 
design  a  more  testable  product,  since  the  contract  is  awarded  to  the  lowest  bidder  and 
specific  testability  requirements  are  not  part  of  the  contract.  It  is  also  difficult  to  get 
management  to  know  how  well  testability  insertion  is  being  incorporated,  as  the 
testability  audits  frequently  have  little  relationship  to  subsequent  field  false  alarm 
problems,  quality  of  ATE  diagnostic  ability,  etc.  Rather  than  having  the  ATE  "work 
harder"  by  increasing  test  speed  to  decrease  execution  time,  the  electronics  designers 
should  "work  smarter"  ensuring  better  BIT  so  as  to  reduce  the  number  of  box  removals  and 
make  the  ATE  diagnosis  tasks  simpler.  The  economic  leverage  of  eliminating  a  one  month 
trip  through  the  maintenance  pipeline  is  much  greater  than  reducing  the  length  of  a  test 
from  two  hours  to  one  hour. 

3.2.3  Symptomatic  Cures 

The  following  techniques  were  found  practical  to  provide  symptom  relief  for  problems 
that  impact  ATE  through  more  cost  effective  generation  of  the  test  program  set: 
o  Improve  Automatic  Test  Program  Generation  for  complex  circuits 
o  Develop  Knowledge  Based  TPS  Assistant  (Freund  43) 
o  Develop  an  AI  Assistant  for  Test  Program  Set  Verification  and  Validation 


However,  the  evaluation  portion  of  this  study  (see  Section  6.1.1,  Life  Cycle  Cost 
Evaluation  Criteria)  found  that  reducing  the  problem  cause  is  more  cost  effective  than 
trying  to  provide  symptom  relief  at  each  development  phase. 

3.2.4  ATE  Problem  Solutions 

ATE  problem  solutions  involve  both  enforcement  of  testability  standards  and  the  use  of 
AI.  The  proposed  military  standard  on  testability  (21)  will  make  the  contractors  aware  of 
test  needs  by  requiring  them  to  do  specific  planning,  tasks,  audits,  etc.  However  care 
must  also  be  taken  to  balance  additional  administrative  overhead  in  a  program  with  good 
design  practice.  AI  provides  a  means  to  create  tools  to  be  used  by  industry  for  testability 
insertion  BIT  design,  and  maintenance  expert  design.  It  will  give  the  designer  leverage  in 
his  attempt  to  "design  smarter."  These  ideas  are  expanded  in  the  AI  applications 
presented  in  Section  5.0. 


4.0  SURVEY  OF  TESTABILITY  APPLICATIONS 


This  survey  focused  on  those  applications  of  artificial  intelligence  that  have  the  potential 
of  leading  to  quick  payoffs  and  practical  use.  Six  testability  improvement  areas  were 
found  to  have  a  potential  for  near  term  application  with,  or  as  part  of  an  AI  approach. 
These  six  testability  areas  are  summarized  in  this  section. 

1)  Self  improving  diagnostics: 

o  Functional  test  sequences  can  be  cost  effectively  improved 

o  Incorporation  of  diagnostic  experience  through  automated  learning  is  a 
promising  near-term  concept 

o  AI  cannot  improve  observability  of  tested  element 

2)  More  effective  fault  detection  and  isolation: 

o  Better  Design  For  Testability  is  the  most  cost  effective  means 

o  Smart  BIT  provides  the  only  means  to  capture  fault  data  in  the  field 

3)  Discrimination  between  false  alarms  and  intermittent  faults: 

o  This  capability  will  take  at  least  five  years  to  evolve.  Information 
necessary  is  not  captured  and  AI  analysis  techniques  are  not  yet 
developed. 

4)  Reduction  of  skills  required  for  maintenance: 

o  AI  applied  to  information  processing  can  reduce  specific  experience 
needed  by  a  maintenance  technician 

o  Computer  Aided  Instruction  has  not  been  proven  to  significantly  improve 
learning 

5)  Integrated  Diagnostics: 

o  AI  can  provide  cost  effective  information  processing  for  system,  ATE, 
and  bench  testing 

6)  Design  For  Testability: 

o  Testability  expert  (human  or  machine)  is  needed  during  the  design 


process 


4.1  SELF  IMPROVING  DIAGNOSTICS 


Fault  diagnosis  consists  of  three  phases— fault  detection,  fault  verification  and  fault 
isolation.  The  criteria  for  a  self  improving  diagnostic  (SID)  system  are  as  follows: 

o  The  system  must  be  able  to  detect  a  fault,  verify  if  it  is  a  hard  fault,  (not  a 
false  alarm  or  intermittent)  and  isolate  the  fault  to  the  replaceable 
component. 

o  It  must  recommend  and  remember  a  fix. 

o  After  the  recommended  fix  is  made,  it  must  determine  correctness, 

o  If  the  recommended  fix  is  correct,  then  the  information  is  saved.  If  the  fix  is 

not  correct  then  the  system  must  try  a  new  fault  isolation  strategy.  When  the 
correct  fix  is  found  for  the  fault,  this  fix  is  saved,  and  the  fault  isolation 
strategy  is  updated. 

4.1.1  Step  One:  Adapt  Functional  Test  Sequences 

Frequently  a  military  ATE  test  will  consist  of  a  fixed  sequence  of  tests.  The  first  form  of 
improvement  is  to  allow  the  functional  test  sequence  to  adapt  to  field  testing  experience. 
Simpson  (29)  has  proven  its  capability  to  learn  the  cost,  time  and  MTBF  associated  with 
test  steps  and  incorporate  this  knowledge  into  optimal  modification  of  the  test  sequence. 
The  knowledge  gained  by  Simpson's  System  Testability  and  Maintenance  Program  (STAMP) 
(29)  observation  of  field  testing  allows  it  to  test  earlier  in  the  test  process,  those  portions 
of  a  unit  which  fail  more  frequently,  thus  decreasing  average  testing  time.  STAMP  also 
has  the  capability  to  optimize  Its  test  sequences  for  either  peacetime  conditions  (low 
cost),  or  war  (quick  response). 

4.1.2  Step  Two:  Incorporate  Diagnostic  Experience  through  Automated  Learning  (IDEAL) 

The  concept  of  IDEAL,  proposed  by  this  study,  is  to  capture  enough  diagnostic  information 
from  the  field  to  allow  its  automatic  assimilation  by  a  central  processor.  The  diagnostic 
concepts  come  solely  from  the  technician,  and  the  program  has  no  capability  to  reason 
about  the  circuit  function.  IDEAL  is  similar  to  the  "The  'Experience  Algorithm'  in 
Automating  Isolation  of  Single  and  Multiple  Faults,"  (8.2  -  Maintenance  Expert),  in  which 


Arington  defines  experience  as  the  accumulation  of  knowledge  and  skill  through  action. 
He  states  that  "experience  programs  are  not  meant  to  replace  or  even  reduce  fault 
isolation  algorithms  or  the  need  for  testability,  but  they  can  enhance  and  complement 
them  technically  and  economically." 

IDEAL  allows  the  actions  taken  by  expert  technicians  to  provide  guidance  for  all 
technicians.  Pau  (sec.  2.4)  shows  a  tabular  structure  for  describing  circuits  similar  to 
Failure  Modes  and  Effects  Analysis  (FMEA).  This  type  of  structure  would  permit 
implementation  of  IDEAL.  Both  IDEAL  and  the  "Experience  Algorithm"  are  concepts 
which  have  not  been  implemented,  whereas  Dr.  Pau's  work  has.  The  IDEAL  concept  is 
discussed  in  Section  5.8  and  is  worth  pursuing  in  more  detail  for  its  potential  benefit  in  a 
three  to  five  year  time  period. 

4.1.3  Benefits  of  Self  Improving  Diagnostics 

There  can  be  considerable  savings  in  maintenance  time  by  incorporating  the  field 
diagnostic  knowledge.  One  of  the  reasons  for  the  continuing  high  cost  of  fault 
detection/isolation  is  that  real  symptoms  are  frequently  not  identified  in  test  procedures. 
The  benefit  of  adapting  test  sequences  and  IDEAL  is  a  decrease  in  the  average 
maintenance  time  (functional  plus  diagnostic  test  time)  for  all  personnel. 

However,  figure  4.1-1  shows  that  the  experts  which  make  up  the  top  10%  (Decile  #1)  of 
the  workers  are  correct  three  times  as  often  as  the  average  worker.  If  am  expert  system 
could  be  used  by  everyone  that  had  one  half  the  performance  of  the  top  experts,  then  the 
average  shown  in  the  figure  would  increase  by  50%. 

4.1.4  Status 

Naval  Electronic  Systems  is  funding  the  only  known  work  in  self  improving  diagnostics. 
Harris  Corporation  is  under  contract  to  develop  integrated  diagnostics  for  a  portion  of  the 
Tomahawk  Launch  Control  system.  During  1984  the  "A"  specification  will  be  developed, 
and  1985  will  see  the  development  of  hardware  and  software  for  automatic  and  semi¬ 
automatic  diagnosis.  This  one-half  million  dollar  contract  will  use  self  improving 
diagnostic  concepts  to  develop  a  historical  database  of  past  failures  and  their  proper 
solutions.  The  data  will  be  searched  before  exhaustive  tests  are  made  on  faulty  boxes. 


DECILES 


From  "AI  in  the  Airland  Battle",  Astronautics  and  Aeronautics  Duly/Aug  83 
Figure  4.1-1  Distribution  of  Expertise 


This  should  reduce  the  test  time,  and  perhaps  eliminate  the  need  for  a  maintenance 
manual.  However,  we  do  not  expect  results  of  this  demonstration  to  be  available  for 
general  use  for  several  years. 

4.1.5  Limitations  of  Self  Improving  Diagnostics 

The  diagnostics  capability  will  depend  upon  the  testability  of  the  unit  being  diagnosed.  If 
design  for  testability  (DFT)  has  been  ignored  then  fault  isolation  will  be  limited  by  the 
quality  of  the  implemented  DFT;  SID  related  improvement  may  be  small.  DFT  will  also 
limit  improvements  in  fault  detection  and  isolation. 


*.2  MORE  EFFECTIVE  FAULT  DETECTION  AND  ISOLATION 


The  Operations  Reliability/Maintainability  Engineering  Study  conducted  for  the  Army  (39) 
identified  three  reasons  for  the  continuing  high  cost  of  fault  detection/isolation  that  could 
be  reduced  by  an  improved  BIT  capability. 

1.  The  cause  of  approximately  33%  of  all  symptoms  was  found  in  an  aircraft 
system  other  than  the  one  in  which  the  symptom  was  observed. 

2.  Approximately  50%  of  all  symptoms  observed  via  aircraft  instruments  and 
warning  devices  indicated  a  failure  of  the  instrument  or  warning  device  rather 
than  a  failure  of  the  monitored  system. 

3.  Approximately  66%  of  all  symptoms  were  not  found  in  the  system  trouble¬ 
shooting  tables.  In  only  50%  of  the  other  cases,  was  the  most  frequently 
reported  cause  of  a  symptom  listed  among  the  possible  causes  in  the  technical 
manuals. 

Development  of  more  effective  fault  detection  and  isolation  techniques  could,  on  the 
surface,  help  reduce  the  impact  of  three  major  testability  problem  areas:  lack  of  design 
for  testability  (DFT),  inadequate  test  procedures  and  inadequate  test  equipment.  These 
are  not  totally  independent  problems,  however.  Resolution  of  those  problem  areas  will, 
for  example,  likely  involve  recommendations  for  use  of  specific  DFT  approaches  that  in 
turn  aid  test  procedures  and  simplify  test  equipment  requirements.  It  has  been  shov-  that 
the  complexity  of  military  electronics  must  be  lowered  to  achieve  reduction  in  repair 
time  (7). 

One  typical  problem  in  fault  detection  and  isolation  is  the  localization  of  faults  in  the 
conformally  coated  printed  circuit  cards  commonly  used  in  avionics.  Most  techniques  in 
use  require  the  unit  to  be  probed.  These  require  recoating  at  best,  and  risk  card  damage 
at  the  worst.  An  alternative  to  probing  is  to  generate  a  fault  matrix  to  allow  fault 
isolation  based  on  a  set  of  fault  signatures,  however  it  is  frequently  not  possible  to  store 
all  possible  fault  matrices.  AI  may  aid  in  the  development  of  simulating  the  faults  while 
the  test  is  in  progress.  Present  work  on  expert  systems  at  MIT  and  Stanford  is  in  the  area 
of  causal  knowledge  in  which  AI  aids  in  modeling  the  operation  and  function  of  a  unit. 
Recent  work  by  Davis  at  MIT,  applies  causal  modeling  to  diagnostics  (27).  In  his  approach, 


most  of  the  knowledge  that  would  be  needed  by  a  technician  to  diagnose  a  problem  is 
included  as  part  of  the  expert  system.  Work  in  this  area  is  also  being  done  by  Genesereth 
(26)  in  his  DART  project. 

The  usefulness  of  causal  knowledge  is  in  dispute  by  some  of  the  AI  community,  but  the 
general  consensus  is  that  it  will  play  a  large  part  in  future  expert  systems.  There  is  some 
risk  however,  because  no  concrete  evidence  has  been  shown  that  the  use  of  causal 
knowledge  will  be  practical  within  the  next  three  to  five  year  time  frame. 

*.3  DISCRIMINATION  BETWEEN  FALSE  ALARMS  AND  INTERMITTENT  FAULTS 

Discrimination  between  false  alarms  and  real  faults  is  one  of  the  toughest  and  most 
controversial  aspects  of  testing.  The  definition  of  false  alarm  is  "an  indicated  fault  where 
no  fault  exists"  (see  glossary,  9.0).  The  Integrated  Test  and  Maintenance  study  done  for 
AFWAL  (1)  concluded  that  in  order  "to  eliminate  the  false  alarms  due  to  design  defects, 
the  only  solution  is  a  design  and  test  effort."  Modifications  to  both  the  operational 
hardware  and  its  BIT  during  full  scale  development  as  part  of  the  design  process  of  a 
program  can  eliminate  a  majority  of  these  false  alarms.  During  full  scale  development  it 
would  be  more  appropriate  to  refer  to  those  alarms  as  design  deficiency  alarms,  and  only 
after  that  phase  refer  to  them  as  false  alarms.  False  alarms  caused  by  environmental 
conditions  such  as  voltage  transients,  electromagnetic  interference,  dust,  background 
radiation,  side-effects  of  other  system  failures,  or  operator  error  are  defects  which  can 
often  be  eliminated  during  initial  testing. 

AI  approaches  have  no  magical  techniques  for  discriminating  between  false  alarms  and 
intermittents.  AI  cannot  go  beyond  the  information  provided  by  people  and  field  data. 
The  Integrated  Test  and  Maintenance  project  (i)  proposed  a  fault  classification  expert 
system  that  would  discriminate  between  false  alarms  and  intermittents.  AI  provides  a 
means  of  acquiring  the  field  diagnostic  experience  and  organizing  it  for  cost  effective 
use.  With  this  field  experience  information,  a  determination  can  be  made  if  similar  units 
will  give  a  false  alarm  when  presented  with  an  identical  environment.  Intermittents,  on 
the  other  hand,  generally  occur  only  for  a  particular  unit  and  frequently  evolve  into  hard 
faults.  In  a  discussion  on  Maintenance  Expert  Evaluation  in  section  5.5  (figure  5.5-2)  we 


predict  the  capability  to  descriminate  will  not  come  until  maintenance  experts  become 
smart  in  their  ability  to  use  the  field  information  and  improvements  in  DFT  and  Smart 
BIT  are  made.  This  will  take  at  least  five  years. 

4.4  REDUCING  SKILLS  REQUIRED  FOR  MAINTENANCE 

The  Air  Force  has  identified  a  need  to  reduce  the  level  of  skills  required  of  maintenance 
personnel.  One  problem  is  because  more  systems  are  being  developed  directly  using 
computers,  they  are  difficult  to  understand  without  also  using  a  computer.  Another 
problem  is  that  in  recent  years  only  relatively  less  skilled  military  personnel  have  been 
typically  available  due  to  the  difficulty  in  recruiting,  training,  and  retaining  qualified 
people.  Maintenance  expert  consulting  systems  and  computer-aided  instruction  systems 
are  two  solutions  normally  considered  for  reducing  required  skill  levels.  But  because  of 
increased  system  complexity  the  ability  to  reduce  skill  levels  is  largely  a  myth;  even  the 
best  test  capability  will  not  solve  100%  of  the  test  and  maintenance  problems.  Skilled 
technicians  will  still  be  needed  for  the  tough  problems,  even  if  they  occur  less  frequently. 

4.4.1  Maintenance  Expert  Systems  Reduce  Required  Experience 

Current  military  ATE  systems  require  that  technicians  acquire  a  great  deal  of  specific 
experience  for  each  item  they  are  to  diagnose.  This  is  acceptable  for  high  volume 
commercial  testing  where  a  technician  maintains  only  a  few  items,  and  thus  can  gain  the 
diagnostic  experience  in  a  few  months.  However,  a  military  technician  who  must 
maintain  hundreds  of  different  items  using  ATE  will  require  several  years  to  gain  much 
experience  on  most  items,  at  which  point  his  term  of  duty  may  be  over. 

By  providing  the  technician  with  easy  access  to  past  information  presented  to  him  in  a 
relevant  fashion  for  the  specific  box  being  tested,  he  can  become  efficient  on  his  job 
much  more  quickly.  The  types  of  information  that  the  AI  applications  described  in  this 
report  can  provide  are: 

o  Information  concerning  the  field  fault  history  of  the  unit 

o  Information  on  past  similar  faults  and  their  diagnosis 

o  An  up-to-date  schematic  with  related  fault  symptom  information  indicated 
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o  Feedback  of  correctness  of  his  diagnosis 


If  a  specific  fault  with  successful  diagnosis  has  occurred  before,  the  maintenance  expert 
system  would  suggest  that  the  same  sequence  of  functional  and  diagnosis  tests  be 
performed  again.  It's  worth  pointing  out,  however,  that  although  a  number  of  expert 
systems  and  training  systems  have  been  built  to  aid  in  diagnostics  of  electrical  and 
mechanical  devices,  none  surveyed  were  designed  to  reduce  the  skills  required  for 
maintenance  in  the  military. 

Boeing  Aerospace  Company  research  on  the  application  of  expert  systems  to  maintenance 
has  led  to  a  number  of  conclusions  regarding  maintenance  personnel.  First,  in  order  to 
attain  mission  readiness,  a  maintenance  technician  requires  a  number  of  skills  such  as 
interpretation  of  maintenance  manuals,  choosing  the  correct  tool  for  the  task  and  being 
able  to  diagnose  equipment  faults  effectively.  Accomplishing  these  tasks  takes  not  only 
ability,  but  time.  Second,  most  expert  systems  presently  in  use  assist  the  user  only  in  a 
single  domain,  such  as  diagnostics.  In  addition,  they  do  not  give  sufficient  help  at  the  skill 
level  of  the  military  user.  Boeing  research  found  that  what  is  needed  is  a  multidomain 
expert  system  that  offers  diagnostic  capabilities,  maintenance  procedures,  data  collec¬ 
tion,  data  analysis  and  help.  At  this  time,  Boeing  Aerospace  Company  is  conducting 
applied  research  on  this  type  of  a  system;  a  prototype  of  the  Maintenance  and  Diagnostic 
Information  System  (MDIS)  is  being  built  (37),  and  the  concept  has  been  shown  to  be 
feasible  (see  Section  2.10). 

4.4.2  Computer-Aided  Instruction  (CAI) 

Publications  on  computer-aided  learning  have  shown  CAI  is  not  able  to  produce  any 
significant  cost  savings  in  training  compared  to  current  techniques.  Even  with  the  use  of 
video-discs  there  is  not  a  significant  increase  in  the  amount  of  information  retained  or  a 
decrease  in  the  training  time  over  conventional  classroom  techniques  (40). 

In  a  related  approach,  research  has  shown  that  it  is  feasible  to  have  a  maintenance  expert 
system  adapt  to  each  user  and  determine  his  or  her  specific  learning  problems  by 
analyzing  responses.  The  system  would  then  alter  its  training  strategy  to  fit  the 
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particular  user.  Unfortunately,  it  does  not  appear  that  this  type  of  system  will  be  ready 
in  the  near  term  though  the  ability  to  adapt  learning  to  each  use  is  being  aggressively 
researched  by  many  organizations  (41).  Practical  development  is  at  least  five  years  away. 

4.4.3  Conclusion 

In  the  near  term,  AI  cannot  significantly  improve  pre-job  training  or  perform  diagnostic 
isolation;  however,  AI  can  now  automatically  acquire  diagnostic  experience  from  expert 
technicians  and  provide  an  integrated  information  display  to  all  technicians.  While  not 
reducing  the  skill  required,  AI  can  reduce  the  amount  of  unit  specific  experience  required. 
This  should  make  inexperienced  technicians  more  productive,  their  job  less  frustrating  and 
more  enjoyable,  which  may  increase  how  long  they  stay  on  the  assignments. 

4.5  INTEGRATED  DIAGNOSTICS 

All  military  services  are  very  concerned  about  the  concurrent  trends  of  increasing 
electronics  complexity  and  the  decreasing  skill  level  of  technicians.  ATE  has  tended  to 
remove  the  technician's  intellectual  capability  from  the  testing  process,  so  that  his  job  is 
now  very  routine,  and  does  not  require  him  to  "zero-in"  on  the  problem.  However,  ATE 
still  requires  highly  trained  specialists  to  isolate  the  problems  ATE  can't  diagnose,  usually 
false  alarms  and  inter  mittents. 

After  several  years  of  study  and  meetings  a  program  office  has  been  set  up  for  integrated 
diagnostics.  The  Integrated  Diagnostics  Program  Office  (ASD/AEGA  at  Wright-Patterson 
AFB)  is  very  concerned  that  technicians  be  given  some  maintenance  aids  for  all  types  of 
electronics  faults.  Now  automatic  testing  in  the  system  and  ATE  usually  have  hard  fault 
diagnosis  coverage  in  excess  of  90%,  but  the  most  difficult  diagnostic  problems  are  still 
left  to  the  maintenance  technican.  The  problems  of  false  alarms  and  intermittent  faults 
give  the  technician  an  additional  diagnostic  burden.  Non-AI  techniques  to  improve  auto¬ 
matic  tests  by  incorporation  of  diagnostic  experience  are  so  manpower  intensive  that  they 
are  not  cost  effective,  and  are  therefore  rarely  used. 
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AI  techniques  will  aid  integrated  diagnostics  by  providing  tools  to  process  information  for 
use  in  design,  operations,  and  maintenance.  Artificial  Intelligence  presently  does  not  have 
the  capacity  to  'think'  in  the  classical  sense,  but  can  process  information  in  the  following 
ways: 

o  Logical  acquisition  of  data 

o  Data  analysis  (pattern  recognition) 

o  Intelligent  retrieval  and  display  (including  graphics)  from  database 

If  the  information  is  not  available  in  a  given  situation,  Al  cannot  create  it.  AI  will  not  be 
able  to  model  the  details  of  circuit  operation  until  the  late  80's  (interview  with  Davis, 
Genesereth),  but  it  can  now  record  human  diagnostic  experiences  and  provide  some 
generality  to  them  so  that  when  "similar"  situations  occur,  the  technician  will  be  informed 
of  the  previous  diagnostic  experience.  This  is  of  some  help  for  the  expert  technician  who 
has  a  lot  of  diagnostic  experience  to  draw  upon,  but  its  primary  utility  is  for  the  average 
technician  who  does  not  have  decades  of  general  experience  and  years  of  system  specific 
experience.  In  all  near  term  applications,  AI  aids  the  technician  in  the  performance  of  his 
task,  but  does  not  replace  him.  A  maintenance  expert  system  would  be  expected  to 
improve  the  performance  of  all  but  the  best  experts  (see  figure  4.1-1). 

The  maintenance  expert  system  provides  diagnostic  experience  to  the  technician  in  the 
form  of  suggestions  in  context  of  the  test  in  progress.  The  maintenance  expert  should 
graphically  display  the  history  of  the  current  unit  under  test,  and  from  its  diagnostic 
experience  database  show  a  similar  situation  and  what  was  done  by  the  person  at  that 
time. 

AI  provides  a  cost  effective  means  to  incorporate  the  diagnostic  experience  for  system 
testing,  ATE,  and  bench  testing.  Integrated  Diagnostics  thus  far  has  considered  only  two 
categories  of  tests:  automatic  and  manual.  The  Smart  Bench  maintenance  expert,  which 
performs  functional  and  diagnostic  testing  with  bench  test  equipment,  adds  a  new 
category  between  those  two.  It  is  targeted  for  those  units  normally  tested  on  the  bench 
or  that  are  marginally  acceptable  for  ATE  testing.  It  provides  a  method  to  capture 
detailed  functional  and  diagnostic  information  from  bench  testing.  (See  Section  5.8) 


4.6  DESIGN  FOR  TESTABILITY 
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Testability,  as  it  now  stands  in  the  industry,  is  a  "bottom  up"  process.  Virtually  all  the 
known  techniques  require  a  detailed  circuit  design  before  a  system's  testability  can  be 
calculated.  However,  design  is  a  "top  down"  process.  The  designer  is  given  requirements 
for  design,  creates  a  design  approach,  analyzes  it  for  performance,  selects  an  architec¬ 
ture,  defines  subsystem  structures  and  performance  requirements,  and,  finally,  evolves  a 
structural  design  that  can  be  analyzed  for  testability.  It  is  very  difficult  for  an  electronic 
designer  to  consider  all  of  the  testability  requirements  of  a  given  product.  The  knowledge 
needed  requires  a  testability  expert  to  be  available  during  the  design  process  to  consult 
with  the  designer.  Table  4.6-1  shows  some  of  the  testability  knowledge  that  this  study 
found  is  needed  during  design.  A  methodology  is  needed  to  handle  DFT  techniques  from 
the  top  down.  One  approach  is  described  in  "Computer  Aided  Testability,"  (2)  where  the 
potential  use  of  AI  techniques  was  mentioned! 

"A  number  of  terms  have  been  developed  to  describe  the  various  levels  of  a  design  in 
going  from  subsystem  to  detailed  level.  The  top  level  is  referred  to  as  the 
"behavioral"  level.  Typically,  at  this  point,  the  design  has  not  been  committed  to 
hardware  and  the  prime  emphasis  is  performance.  The  next  level  has  been  described 
as  "functional"  or  more  recently  "architectural."  At  this  level,  a  particular 
implementation  has  been  defined—  analog,  digital,  mechanical,  etc.  Within  the 
architectural  level  there  are  further  subdivisions.  "Microarchitecture"  has  been 
coined  to  define  the  top  level  structural  details,  typically  in  a  register  level  form. 
A  lower  level  description  in  this  same  level  is  the  "logical"  description  in  the  form 
of  schematics  that  have  not  been  committed  to  specific  parts.  The  .~west  level, 
excluding  semiconductor  design,  is  the  "structural"  level  (or  "physical")  which 
defines  in  detail  the  specific  manufacturing  process,  e.g.,  interconnects,  parts,  etc. 
Typically  it  is  at  this  level  that  most  testability  analyses  are  currently  performed 
today.  In  effect,  these  analyses  are  after  the  fact  audits,  and  DFT  is  often  ignored." 
(See  figure  4.6-1) 
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Knowledge  Needed  Requires  Testability  Expert 


Military  system  requirements 

•  Short  test  times  requirements 

•  Fault  tolerance  requirements 

•  Availability 

•  Production 

•  Operation 

Factory  production/test  generation 

•  ATE:  Functional,  in-circuit^bench 
Maintenance/reliability 

•  System  availability 

•  Maintenance  cycle,  experience  level  of  technicians 

•  Integrated  diagnostics 

•  Diagnostic  concept  BIT/ ATE  trade 

•  Bit  selection  (dozens  of  types) 

•  Must  diagnose  all  faults  at  all  levels  of  maintenance 

•  Data  collection  and  test  improvement 

•  Effect  of  intermittents  -  false  alarm  rate 

Corporate  design  rules  (LSSD,  etc) 

Costs  of  testability 

•  Weight  -  volume 

•  Memory  constraint 

•  Production  costs 

•  Reliability  degradation  due  to  additional  circuitry 


Table  4.6- 1  Testability  Requirements 
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Figure  4.6-1  Design  for  Testability  Drawing  Board  (49) 


A  potential  AI  solution  to  this  DFT  problem  is  covered  by  the  computer  aided  preliminary 
design  for  testability  opportunity  described  in  section  5.3.  The  RADC  testability 
notebook  shows  (figures  4.6-2  and  4.6  3)  that  testability  incorporation  during  preliminary 
design  has  the  largest  impact  (70%)  on  life  cycle  costs  so  this  study  is  evaluating  the  use 
of  preliminary  rather  than  the  typical  post  DFT  aides 
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"Intensive  total  system  design  analyses  are  essential 
for  lowest  operations  and  support  cost,  by  proper 
design  for  PM,  FD/FL  (BIT/BITE)  integral  with  system 
(70%  impact  potential)". 


Figure  4.6-3  Opportunity  for  Cost  Improvement  (22) 


5.0  POTENTIAL  AI  SOLUTIONS  TO  TESTABILITY  PROBLEMS 


5.1  INTRODUCTION 

The  survey  of  AI  techniques,  ATE/BIT,  and  testability  applications  permitted  considera¬ 
tion  of  many  possible  AI  applications  to  testability.  The  many  factors  affecting  the 
desirability  of  an  AI  solution  are  discussed  in  section  5.2.  The  applications  are  coalesced 
into  specific  solutions  (opportunities)  which  are  discussed  in  sections  5.3  to  5.9,  often  with 
the  aid  of  figures  predicting  their  evolutionary  development.  The  requirements  for  the 
development  and  use  of  the  AI  applications  are  met  by  the  engineering  work  station  (see 
5.10).  The  detailed  evaluation  of  the  solutions  follow  in  section  6.0. 

Some  key  findings  of  this  section  are: 

o  Computer  aided  preliminary  design  for  testability  can  create  the  testability 
expert  needed  for  early  phase  DFT  reviews 
o  Smart  BIT— first  needs  to  record  field  fault  data,  which  can  be  done  with  a 
programmable-recording  BIT  chip  set,  and  then  evolve  into  incorporation  of  AI 
techniques 

o  Maintenance  expert  systems 

o  cost  effective  now;  will  increase  their  ineffectiveness  as  tools  (rules)  are 
developed  to  ease  their  creation 

o  evolutionary  development  will  greatly  increase  their  capability 
o  testability  improvements  are  necessary  to  make  full  use  of  the  increased 
capability 

o  maintenance  experts  for  ATE,  bench,  system  used  on-line  and  off-line 
share  many  development  features,  but  have  different  benefits  and  risks, 
o  Engineering  work  stations  meet  the  requirements  for  development  and  use  of 
the  solutions 


5.2  INITIAL  EVALUATION  OF  AI  APPLICATIONS  TO  TESTABILITY 


Since  the  goal  of  this  study  was  to  find  practical  and  cost  effective  applications  of  AI  to 
testability,  those  applications  identified  during  the  survey  phase,  with  little  potential, 
were  eliminated.  The  development  of  testability  needs  and  a  listing  of  applications 
rejected  is  covered  in  this  section.  The  cause  of  rejection  generally  was  due  to  the 
necessity  of  waiting  for  a  heavily  funded  AI  research  effort  to  develop  a  cost  effective 
tool  or  because  the  application  would  have  a  very  small  market. 

5.2.1  Military  Testability  Needs  Are  Emphasized 

Military  and  commercial  testability  needs  differ  in  many  ways.  Some  AI  applications, 
while  being  useful  for  commercial  applications  would  not  be  cost  effective  or  desirable 
for  military  products.  Military  testability  has  certain  requirements: 

o  Low  volume  production 

o  High  stress  environment  that  results  in  increased  intermittents 
o  High  complexity  yet  restricted  test  access  due  to  conformal  coating 
o  Distributed  repair  by  low  skill/short  longevity  personnel 
o  Old  ATE  with  slow  and  limited  ability  test  programs 
o  Fault  tolerance  and  BIT  used  more  than  by  commercial  products. 

Commercial  testability  needs  are  different  and  include: 

o  High  volume  production, 

o  Need  for  rapid  testing  on  production  line 
o  Use  of  VLSI  in  designs 

o  Use  of  state-of-the-art  components  and  ATE 

o  VLSI  Design  must  be  correct  the  first  time  due  to  competitive  market 
o  Production  line  testability  emphasis  rather  than  field 
o  Test  and  repair  by  board  swap  is  extensively  used 

(Field  service  is  not  as  concerned  with  component  isolation) 
o  More  benign  operating  environment  (except  for  automotive  electronics) 

These  differences  need  to  be  considered  when  judging  the  practicality  of  applying  AI  to 
military  testability  needs. 
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5.2.2  What  Makes  an  Application  Desirable 


Applications  which  create  very  unique  or  identifiable  benefits  were  the  prime  considera¬ 
tion  of  this  study,  that  is,  the  application  would: 

o  Perform  jobs  not  previously  done  due  to  lack  of  effective  tools 

o  Result  in  general  purpose  tools  which  can  be  widely  applied 

o  Provide  significant  downstream  benefits 
o  Use  AI  techniques  already  in  use  (and  is  therefore  credible) 
o  Create  or  use  breakthroughs  which  had  the  potential  of  a  10X  improvement 
over  previous  methods  (examples  are  tools  running  on  engineering  work 
stations,  machine  learning,  and  metarules) 

5.2.3  Desirability  Factors 

An  application  will  be  readily  applied  if  it  benefits  all  affected  organizations.  Each  of  the 
applications  finally  recommended  (sec.  7.0)  provide  benefits  (or  at  least  no  cost  increase) 
to  each  of  the  following  four  types  of  organizations: 

o  Military  Projects 

o  Support  Organizations 

o  R&D  Organizations 

o  Contractors  and  Subcontractors 

Figure  5.2-1,  Desirability  of  AI/T  Applications,  shows  the  relationship  of  the  various 
factors  that  affect  whether  the  military  would  want  to  use  the  application.  The  following 
paragraphs  discuss  key  points  of  that  diagram. 

Does  Military  Project  Want  It? 

The  military  project  which  controls  an  item's  production  contract  does  not  want  to  take  a 
risk  with  AI  technology  unless  there  is  significant  change  in  one  of  the  following  three 
items: 

o  Does  it  improve  mission  critical  performance? 
o  Are  acquisition  costs  reduced  significantly? 

o  Will  it  increase  management  visibility  of  testability? 


The  field  support  organizations  have  been  "trained"  by  experience  not  to  trust  BIT 
indications,  due  to  excessive  false  alarm  rates.  The  accuracy  of  maintenance  experts 
must  support  the  requirements.  For  example,  a  flightline  maintenance  expert  should  have 
an  accuracy  of  at  least  90%  if  it  is  to  be  trusted  and  therefore  used. 

Large  organizations  do  not  readily  accept  change  in  their  organizational  structure.  The 
AI  information  flow  should  be  self-contained  so  that  no  organizational  changes  are 
needed. 


R&D;  Practicality  of  Demonstration 

A  major  concern  of  a  government  research  organization  like  Rome  Air  Development 
Center  is  the  risk  of  successful  demonstration  of  the  concept.  One  of  a  research  group's 
major  product  goals  must  be  the  successful  demonstration  of  concepts.  Because  of  this 
need,  the  following  concept  maturity  aspects  were  explicity  considered  in  our  applications 
evaluation: 

o  Time  to  demonstrate 

o  Cost  to  demonstrate 

o  Risk  to  demonstrate 

Do  Contractor  and  Subcontractors  Want  It? 

A  new  testability  technique  should  reduce  the  cost  or  the  time  to  implement  testability 
during  the  concept,  validation,  prototype,  and  production  phases.  Design  for  Testability 
employed  at  the  concept  phase  should  reduce  the  problems  with  all  engineering  develop¬ 
ment  phases. 

Prior  to  the  field  use  it  is  difficult  to  assess  the  adequacy  of  the  DFT  process.  When  the 
testability  requirements  cannot  adequately  be  measured  in  the  early  phases  of  the 
program  (i.e.,  BIT  verification  is  not  easily  performed  for  hard  faults,  much  less 
intermittent  or  real  world  environmental  faults),  extremely  expensive  engineering  change 
proposal  (ECP)  activity  can  result. 


Does  it  improve  Mission  Critical 
Performance  Significantly  (>3%) 


Figure  5.2 - 1  Desirability  ofAI/T Application 


The  contractor  will  hesitate  to  use  a  technique  if  it  requires  him  to  acquire  (or  worse, 
develop)  new  design  support  systems  or  make  major  modifications  to  a  design. 

5.2.4  Potential  AI  Solutions  to  Testability  Problems 

A  matrix  of  the  key  testability  problems  and  the  "mature"  AI  techniques  is  shown  in  table 
5.2-1.  This  summarizes  the  interrelationship  of  these  areas  which  were  separately 
discussed  in  sections  2.0,  3.0,  and  4.0.  It  comes  as  no  surprise  that  expert  systems  are 
very  useful  for  many  of  the  problems.  Machine  Learning  is  an  emerging  AI  technique 
which  can  semiauto matically  create  rules  based  on  experience  in  a  small  fraction  of  the 
time  required  by  rule  based  systems.  Although  natural  language  and  voice  recognition 
would  be  very  useful  AI  techniques  to  be  used  with  expert  systems  etc.,  they  are  not  yet 
ready  to  be  applied;  at  this  point  in  time,  improved  graphics  and  on-screen  manipulation 
techniques  can  provide  a  greater  benefit  than  voice  recognition.  Discussion  of  some  of 
the  analysis  details  that  went  into  the  construction  of  Table  5.2-1  follows. 

There  were  many  suggestions  available  for  solutions  to  improve  testability  using  AI  or 
pseudo-AI  techniques.  Input  was  taken  from: 

o  literature  search 

o  industry  interviews 

o  corporate  communications 
o  statement  of  work  for  this  study 

o  integrated  diagnostics  meetings 

o  AI  conference 

o  ATE  conference 

In  addition  to  the  desirability  factors  (fig.  5.2-1),  this  study  was  concerned  with  the 
practicality  of  development  of  a  solution  during  the  next  three  years  and  it$  payoff 
leverage.  By  far,  the  largest  category  of  potential  solutions  used  maintenance  expert 
systems.  Although  present  versions  are  cost  effective,  the  use  of  breakthroughs  in 
metarules,  machine  learning,  and  engineering  work  stations  can  provide  a  dramatic 
increase  in  capability  and  cost  effectiveness  of  maintenance  experts.  The  wide  variety 
of  these  systems  made  it  difficult  to  lump  their  benefits/risks  into  a  single  solution  for 
description/evaluation.  Therefore,  several  classes  were  created  to  clearly  distinguish 
them.  This  topic  is  discussed  in  detail  in  section  5.5. 


BIT/ATE  FAILURE  COVERAGE  .  X  X 

SELF  IMPROVING  DIAGNOSTICS  .X  X  X 

MORE  EFFECTIVE  FAULT  DETbISOL  .  X  X 

DISCRIMINATING  INTER/FALSE  _ XXX 

REDUCING  FALSE  ALARMS  .  XX 

ISOLATING  INTERMIT  FAILURES  _ XX 

REDUCING  SKILL  LEVEL  FOR  M  .X  XXX 

TESTABILITY  FOM _ . _ XX 

DESIGN/TEST  PROCESS  INTEG.  .  XXX 

CAT  COMPLEXITY  AND  COST _ . _ XXX  X 

DATA  COMPRESSION  AND  MANAGE.  .  XXX 

INTERFACING  <ND  UPWARD  COMPAT.. 

MAN/MACHINE  INTERFACE  .  XX 


X  X  XXX 

ML  ML _ XXX 

XX 

ML _ XXX 

XX 

_ X _ 

ML  X  XX 

_ X _ 

X  X 

x _ m 

XXX 

X 

X 


NOTES 

ML  MACHINE  LEARNING  IS  USED 
N/A  NO  APPLICATION  IN  NEAR  TERM 


APPLICATION/IMPROVEMENT 

X  SOME 

XX  USEFUL 

XXX  VERY  USEFUL 


Table  5.2-1  Al  Techniques  Address 
Testability  Problems 


Of  the  following  potential  A1  solutions  to  testability  problems,  those  items  preqeeded  by 
an  asterisk  were  selected  for  further  analysis  and  are  described  in  more  detail  later  in  this 
section. 

♦  Computer  Aided  Preliminary  Design  for  Testability  (CAPDT)  provides  a  testability 
assistant  directly  available  during  preliminary  design  phases. 

♦  Smart  Built-In  Test  used  in  boxes  or  cards  can  identify  intermittent  faults  and  reduce 
false  alarms. 

»  Smart  System  Integrated  Test  is  a  system  level  Smart  BIT  which  performs  testing  while 
the  system  is  operating. 

♦  Maintenance  Expert  -  Box  provides  ATE  test  management  with  self  improvement  of 
functional  tests. 

♦  Maintenance  Expert  -  System  describes  the  kind  of  capability  that  can  be  expected 
during  the  next  three  years. 

♦  Maintenance  Expert— Smart  incorporates  the  benefits/risks  of  the  evolutionary  insertion 
of  learning  capability  in  the  maintenance  expert— system  and  its  access  to  Smart  BIT 
information. 

♦  Smart  Bench  is  a  maintenance  expert  system  to  semi  automate  bench  testing. 

♦  Automatic  Test  Program  Generation  (ATPG)  can  create  better  functional  tests  if  AI 
provides  ability  to  understand  circuit  functional  operation  level. 

Maintenance  Experts  with  Circuit  Understanding  techniques  is  not  ready;  by  itself  AI  will 
take  at  least  5  years  to  understand  today's  circuits.  This  also  relegates  AI  to  an 
information  manager. 

Robot-ATE  test.  Military  tests  are  so  slow  that  handling  time  is  insignificant.  It  is 
pointless  to  reduce  a  one  minute  handling  time  for  a  one  hour  test.  GenRad  and  Zentel 
have  developed  this  for  commercial  applications  with  production  volumes  of  over  1000  per 
day.  This  AI  application  is  most  applicable  to  high  rate  production  environments. 


Computer  Aided  Instruction  results  in  neglible  cost  saving  at  this  time. 

Natural  Language  Interface  techniques  are  rapidly  developing  and  heavily  funded  by  many 
organizations.  However,  tools  are  not  yet  available  for  use  with  other  AI  techniques  and 
use  of  graphics  is  a  less  risky  way  to  communicate  electronics  information  in  the  near 
term. 

ATE  Replacement  Aids  facilitate  the  transfer  of  tests  from  old  ATE  to  new.  Costs  to 
develop  each  aid  cannot  be  amortized,  so  manual  techniques  remain  more  cost  effective. 

Software  Assistant  for  Test  Program  Development  is  not  cost  effective  in  the  near  term. 
Heavy  funding  currently  is  being  applied  to  the  general  software  assistant;  examining 
testability  utilization  of  it  should  wait  a  few  years  until  the  software  assistant  is  a  well 
defined  product.  Additionally,  the  lack  of  ATE  standard  hardware  and  software  and  a 
means  to  fully  describe  the  unit  under  test  will  make  it  difficult  to  make  a  general 
purpose  test  program  set  (TPS)  assistant. 


The  following  sections  present  concept  details  of  the  AI  applications  selected  for  analysis. 


N  S 


5.3  COMPUTER  AIDED  PRELIMINARY  DESIGN  FOR  TESTABILITY  (CAPDT) 


Computer  aided  preliminary  design  for  testability  (CAPDT)  is  a  testability  expert  which 
can  access  testability  during  early  phases  of  design. 

The  findings  related  to  this  application  are: 

o  Testability  experts  are  needed  during  initial  design  to  meet  new  requirements, 
o  Not  enough  human  testability  experts  are  available, 
o  An  A1  testability  expert  can  be  made, 
o  Research  is  currently  underway  that  supports  the  concept, 
o  Large  benefits  in  total  LCC  as  well  as  decreased  production  phase  cost  are 
possible. 

Development  Status 

To  date,  all  development  in  this  area  has  been  concerned  with  the  manufacture  of  very 
large  scale  integration  of  circuits  (VLSI),  where  the  lack  of  testability  and  self-test  can 
result  in  an  integrated  circuit  slipping  its  schedule  and  thus  loosing  its  position  in  that 
aggressive  market  place.  "A  Merger  of  CAD  and  CAT  is  breaking  the  VLSI  test 
bottleneck"  (35)  and  "Automation  in  Design  for  Testability"  (38)  are  recent  overviews  of 
VLSI  DFT. 

5.3.1  Need  To  Apply  Testability  Expertise  During  Initial  Design 

Manual  and  computer  aided  testability  audits  are  characteristically  applied  too  late  for 
results  to  be  incorporated  into  the  design  and  typically  have  too  narrow  a  focus  (e.g., 
focus  on  only  ATE).  As  a  result,  low  testability  has  caused  high  operations  and 
maintenance  costs  due  to  false  alarms,  intermittent  faults  and  diagnostic  isolation 
ambiguity.  A  testability  expert  working  with  the  designers  during  preliminary  design  can 
provide  suggestions  and  comments  that  will  improve  insertion  of  testability.  To  do  this 
effectively,  the  expert  needs  knowledge  of  the  design  hierarchy  from  the  integrated 
circuit  up  through  system  level  design. 

The  IDA  Test  Technology  Report  pointed  out  the  LCC  reduction  benefits  from  early 
testability  insertion: 
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o  "Designing  in  testability  on  new  weapon  systems  had  the  highest  potential  of 
all  approaches  in  testing  technology  research  and  development.  A  10%  to  20% 
reduction  in  the  life  cycle  costs  can  be  achieved  by  employing  good  testability 
design." 

o  They  also  recommended  that  the  military  "initiate  a  major  weapon  system 
design  technology  program,  which  injects  testing  technology  into  this  design 
process." 

5.3.2  Few  Testability  Experts 

Electronic  system  designers  cannot  be  expected  to  be  aware  of  all  of  the  test 
considerations  which  need  to  be  traded;  it  is  difficult  enough  just  to  understand  the  dozens 
of  testability  requirements  levied  against  a  program  (see  section  4.6).  More  testability 
experts  could  be  trained,  however;  there  are  no  standard  means  to  evaluate  their  work  and 
it  is  very  difficult  for  one  individual  to  have  awareness  of  all  system  levels  test 
requirements  and  trades.  AI  provides  a  way  of  cloning  the  human  testability  experts,  and 
making  them  available  to  designers.  The  resulting  man/machine  system  will  not  be 
overloaded  with  information  the  way  the  human  would. 

5.3.3  Description  of  CAPDT 

Computer  aided  preliminary  design  for  testability  is  a  rule  based  testability  assistant 
which  can  be  developed  for  engineering  work  stations  to  provide  testability 
reviews/assistance,  computer  aided  learning,  and  provide  suggestions  at  the  early  phase  of 
the  system  development  (fig.  5.3-1).  This  testability  assistant  would  be  used  to  improve 
testability  and  thus  produce  cost  reductions  at  all  subsequent  phases  of  system  develop¬ 
ment.  It  would  also  support  electronics  design  at  all  levels  of  implementation  from 
system  design  down  to  integrated  circuit  design. 

In  order  for  CAPDT  to  analyze  a  circuit  it  must  first  be  recognized.  "A  High  Level 
Recognizer  for  Schematic  Diagram"  (IEEE  Transactions  on  Computer  Graphics  and 
Applications  May  1983)  shows  the  progress  in  automatic  circuit  and  network  recognition. 
CAPDT  would,  in  the  near  term,  have  mem-aided  computer  recognition,  where  the 
machine  would  ask  questions  about  functionality  and  testability  of  the  network.  The 


Engineering  Work  Station 


Figure  5.3-1.  Computer  Aided  Preliminary  Design  for  Testability 


following  hypothetical  design  or  interaction  with  DFT  system  is  taken  from  Horstmann. 
"Design  for  Testability  using  Logic  Programming,"  (32k 

Designer  Input  System  Response 

Start  DFT  Check  CLK1  cannot  be  observed. 

What  should  I  do? 

Why  can't  you  observe  CLK1?  Failed  to  connect  this  clock  to  an 

output. 

I  have  added  an  additional  output  (P02) 
for  CLK1. 

CLK2  cannot  be  observed. 

What  should  I  do? 

!  have  added  an  additional  output  (P03) 
for  CLK2. 

The  design  is  correct  as  it  currently 
stands. 

This  interview  session  with  the  "testability  expert"  would  produce  a  tabularized  descrip¬ 
tion  of  the  network  attributes  similar  to  FMEA  and  Dr.  Pau's  work  (see  sec  2.4). 

CAPDT  would  have  several  classes  of  resources  to  draw  upon  using  intelligent  retrieval 
techniques: 

o  Design  rules  for  BIT,  Fault  Tolerance 
o  Testability  estimators 

o  LCC  estimators 

o  Information  Base  of  previously  analyzed  examples 

o  Diagnostic  metarules 

Testability  information  can  also  be  provided  to  the  designer  in  the  form  of  examples  of 
test  strategies  previously  used  for  similar  requirements/constraints.  Installing  a  testabil¬ 
ity  tutor  of  this  type  on  an  engineering  workstation  can  provide  a  very  powerful  teaching 
tool.  By  having  a  very  large  database  which  can  be  altered,  it  can  predict  the  effect  in 
change  of  test  strategy.  It  can  have  examples  of  good  and  bad  BIT,  self  test,  reliability 
designs  etc. 


Connect  it. 

Continue  with  check. 

Continue  with  check. 


3.3.*  Status  and  Evolution 

A  form  of  CAPDT  is  being  developed  for  integrated  circuits  in  the  commercial  market 
place  and  for  VLSI  and  the  military  market  place  through  the  VHSIC  development.  Mangir 
(34)  discusses  the  use  of  expert  systems  for  assessing  the  design  tradeoffs  during 
preliminary  design  of  built-in  self-test  for  VLSI. 

Steinberg  (AAAI- 1983)  "An  Intelligent  Aid  for  Circuit  Redesign"  extends  his  work  at 
Rutgers  on  CRITTER  to  evaluate  options  for  changing  a  digital  electronics  design. 
Redesign  has  much  less  scope  than  design,  so  is  much  more  tractable.  The  ability  of  AI  to 
suggest,  then  investigate  the  impact  of  redesign  is  just  what  is  needed  for  CAPDT. 

Section  8.2  has  additional  pertinent  references.  The  CAD  section  shows  IBM  extensive 
work  in  this  area:  "Structured  Design  Verification:  Function  and  Timing".  A  review  is 
given  in  "Prospects  for  Expert  Systems  in  CAD"  (Stefik  83).  The  testability  section  has 
"Design  for  Test  Calculus:  An  Algorithm  for  DFT  Rules  Checking"  which  is  a  one-pass 
algorithm  for  checking  digital  networks,  especially  VLSI  IC's. 

There  will  be  an  evolution  of  the  capability  of  CAPDT  with  the  initial  systems  having  the 
ability  to  impose  design  rule  checks,  perform  testability  allocation,  investigate  con¬ 
trollability  and  observability  and  aid  in  suggesting  and  performing  trades  such  as  self-test 
versus  external  test  capability.  These  systems  will  be  able  to  evolve  in  several  years  to 
include  the  ability  to  describe  software  functioning,  especially  for  self-test  coverage  and 
perform  a  hierarchy  of  testability  allocation  between  the  system,  subsystem,  box,  card, 
and  integrated  circuit  levels  (fig.  5.3-2).  It  will  be  at  least  five  years  until  this  system 
has  the  capability  of  considering  effects  of  intermittents  in  the  testability  coverage. 

5.3.5  Concerns/Risks 

There  are  several  aspects  which  can  limit  the  near  term  use  or  capability  of  CAPDT: 

o  Restricted  usage  due  to  design  station  diversity  (lack  of  standardization) 

Other  design  stations  used  (VAX  etc.) 

Other  modeling  systems  used  during  preliminary  design, 
i.e.,  computer  hardware  description  language 
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o  Near  term  limitations 

Designer  must  describe  function  performed  by  software 
o  Intermittents  difficult  to  deal  with 

o  National  accumulation  of  a  database  will  be  restricted  by  corporate  interests 

As  recommended  later,  in  section  6,  there  should  be  an  active  program  to  standardize  the 
data  interchange  format  for  all  of  the  AI  solutions  to  testability  so  that  the  knowledge 
can  be  shared. 

The  restrictions  on  describing  software  and  intermittents  will  require  use  of  some 
approximation  techniques  to  provide  descriptions  in  the  near  term. 

Knowledge  defines  the  capability  of  CAPDT,  and  more  knowledge  is  obtained  by  having 
more  users  input  information  into  the  same  knowledge  base. 


5A  SMART  BUILT-IN-TEST  (BIT) 

As  described  in  the  RADC  work  statement  for  its  Smart  BIT  procurement,  "Smart  BIT  is  a 
design  concept  capable  of  identifying  intermittent  faults,  reducing  false  alarms,  and 
discriminating  between  the  two.  It  is  being  implemented  in  part  by  the  general  use  of 
artificial  intelligence  techniques."  It  does  not  use  AI  to  the  exlusion  of  normal  BIT 
techniques.  Smart  BIT  can  be  used  with  boxes  or  cards  of  electronics.  Seif  test  at  system 
level  and  chip  level,  while  also  needing  improvement,  has  different  requirements  which 
are  outlined  at  the  end  of  this  section. 

The  Smart  BIT  discussion  will  proceed  as  follows: 

o  A  programmable  recording  BIT  chip  set  is  needed  as  a  minimum, 
o  Smart  BIT  improves  diagnostic  techniques  of  programmable-recording  BIT. 
o  Signal  analysis  similar  to  a  technician's  approach  can  be  used, 
o  Review  of  status  and  risks. 

5 A.  1  Programmable-Recording  BIT  Chip  Set  is  Needed  as  a  Minimum 

The  requirements  for  improved  BIT  capability  discussed  in  section  3.1  can  be  at  least 
partially  satisfied  with  a  microprocessor  based  BIT  which  is  reprogrammable  and  can 
include  environmental  information  along  with  a  fault  record.  Maintenance  technicians 
and  maintenance  expert  systems  have  a  vital  need  for  this  kind  of  data  to  reduce  RTOK 
incidence.  By  providing  good  fault  recording  capability  the  programmable-recording  BIT 
can  also  capture  fault  information  during  design  and  full  scale  development  which  can 
lead  to  the  elimination  of  false  alarm  causes.  False  alarms  can  be  reduced  after  full  scale 
development  by  software  modification  to  the  programmable  BIT.  Even  though  changes  to 
software  or  hardware  after  configuration  control  has  been  established  are  usually  very 
expensive,  it  is  possible  that  the  design  approach  and  previous  verification/validation  of 
the  BIT  will  have  proven  independence  of  test  routines,  thus  allowing  a  simplier  V3cV 
effort. 

At  this  time,  BIT  is  usually  invented  for  each  project  and  is  rarely  programmable.  This 
has,  in  the  past,  resulted  in  most  testing  organizations  not  trusting  BIT.  Improved 
capability  of  programmable-recording  BIT  or  Smart  BIT  must  be  provided  with  good 
interfaces  to  the  designer,  the  user  and  the  maintenance  technician  if  it  is  to  be  accepted 
and  trusted. 


5.4*2  Chip  Set  and  Software  Will  Reduce  Development  Time  and  Cost 

Present  technology  has  the  capability  to  develop  a  programmable- recording  BIT  chip  set 
for  the  military  that  would  occupy  only  a  few  square  inches  of  board  area.  (Within  two 
years  time,  this  can  be  reduced  further  by  use  of  leadless  chip  carrier  technology.)  The 
read  only  memory  chip  can  contain  proven  software  test  routines  that  can  be  used  by  any 
designer,  eliminating  the  need  for  each  person  to  reinvent,  debug,  verify  and  validate 
totally  new  BIT  appraches  for  each  design. 

The  read  only  memory  will  need  to  contain  a  real  time  operating  system  and  a  method  to 
connect  the  independently  developed  routines  together.  The  concept  "pipes?'  of  the  UNIX 
operating  system,  which  allows  the  output  of  one  process  to  directly  flow  into  the  next 
can  be  used  here.  UNIX  permits  independent  pipes  to  be  created  and  tested  which  can 
later  be  joined  together  into  a  needed  configuration.  The  need  for  assembly  language 
programming  by  each  BIT  designer  should  be  virtually  eliminated  by  the  previously 
developed  and  validated  pipes. 

A  major  requirement  of  this  new  BIT  capability  is  that  it  be  able  to  perform  its  own  self 
test.  This  would  include  routines  to  test  its  processor  and  all  types  of  memory,  and  loop 
aromd  a  majority  of  its  input/output  circuitry.  It  should  also  be  designed  to  fail  in  such  a 
way  so  that  the  failure  of  BIT  is  clearly  distinguishable  from  the  failure  of  the  functional 
unit,  in  addition,  a  BIT  failure  must  not  cause  the  failure  of  the  functional  unit. 

5.4.3  Bit  Chip  Set  Description 

Figure  5.4-1  shows  that  there  is  a  variety  of  memory  utilization  in  the  chip  set  using 
different  types  of  memory  chips.  The  resources  are  in  read  only  memory,  and 
implementation  of  the  test  requirements  are  in  non-volatile  (information  stored  is  not  lost 
when  power  is  removed)  memory.  Environmental  information  would  come  from  the 
system  level  and  be  recorded  with  faults  in  the  non-volatile  memory.  (Strike  (4®  has  a 
good  review  of  electrically  erasable-  programmable  read  only  memory  (EEPROM)  for 
avionics  that  fits  this  application.)  The  ATE/Engineering  work  station  connection  to  the 
chip  set  would  provide  for  reading  of  the  fault  data  and  reprogramming. 


5.4.4  Smart  BIT  improves  Diagnostic  Techniques  of  Programmable-Recording  BIT 

AI  can  improve  diagnostic  capability  of  a  programmable-recording  BIT,  providing  the 
potential  for  systems  to  achieve  an  even  lower  level  of  false  alarms  and  be  able  to 
perform  some  fault  isolation  of  intermittents.  Some  of  the  AI  capability  can  be  internal 
to  the  chip  set,  but,  in  the  near  term,  a  majority  of  it  will  be  in  the  engineering 
workstation  because  knowledge  processing  requires  more  memory  than  is  currently 
reasonable  to  put  onto  a  board. 

5.4.5  Detailed  Descriptions:  Browsing  BIT 

One  promising  SMART  BIT  concept  is  that  of  a  roving  signal  analysis  which  we  have 
termed  "Browsing  BIT".  It  has  the  potential  to  reduce  the  amount  of  dedicated  resources, 
but,  upon  demand,  can  focus  its  attention  on  circuits  having  intermittents.  The  "Browsing 
BIT"  surveys  signal  quality  in  a  semiautonomous  manner.  It  would  be  able  to  detect  a 
substantial,  abrupt  or  irregular  change  in  the  condition  of  the  signals  monitored  to  aid  in 
discrimination  between  false  alarms  and  intermittents. 

The  browsing  BIT  would  have  many  of  the  capabilities  of  a  nonexpert  electronic 
technician  instructed  to  monitor  all  of  the  inputs  and  outputs  to  see  if  they  meet  their 
specifications.  They  both  could,  for  example,  report  when  the  signals  are  degraded.  The 
Browsing  Bit  would  perform  these  tasks  in  places  where  the  technician  could  not  be 
present,  e.g.,  inside  a  missile  in  flight. 

ATE  tends  to  ignore  signal  quality  and  intermittents  which  are  the  focus  of  Browsing  BIT. 
Pulse  signals,  for  example,  could  be  monitored  for  amplitude,  noise,  and  multiple  or  sharp 
pulse  in  addition  to  the  standard  duration  measurement. 

Browsing  BIT  would  have  many  means  at  its  disposal  to  test  signals  and  their  interrela¬ 
tionships.  To  better  understand  what  is  and  is  not  a  failure,  it  would  have  an  ability  to  set 
its  internal  "Browsing"  tolerances  tighter  than  its  alarm  tolerances.  By  doing  this  it  would 
acquire  more  information  about  the  characteristics  and  operating  conditions  associated 
with  each  fault  or  signal  degradation.  This  would  permit  the  future  prognostic  capabilty 
shown  at  the  far  right  of  the  BIT  evolutionary  chart  fig.  5.4-2.  A  Browsing  BIT  would 
spend  varying  amounts  of  time  on  each  of  the  following  four  functions  depending  on 
recent  fault  history: 


o  Test  that  each  parameter  is  within  functional  specification 
o  Monitor  quality  of  all  signals 

o  If  signal  deterioration  is  observed,  increase  monitoring  all  signals  that  could 
have  affect  on  it 

o  Acquire  good  signal  characteristics  and  record  in  nonvolatile  memory  for 
external  analysis  during  program  development  and  verification 

While  the  "Browsing  BIT"  does  not  monitor  all  signals  100%  of  the  time  and  can  miss 
detecting  some  intermittents,  the  ability  to  force  the  unit  to  spend  more  time  on  critical 
circuits  or  likely  fault  areas  as  well  as  its  ability  to  concentrate  on  these  areas  when 
faults  occur,  permit  a  significant  increase  in  observability  of  likely  fault  conditions. 

5.4.6  Chip  Set  Signal  Analysis  Tools  Are  Modeled  After  Technicians  Tools 

A  combination  of  hardware  and  software  signal  processing  modeled  on  a  technician's 
signal  processing  testing  capability  can  provide  a  powerful  set  of  programmable  analog 
circuit  test  capability. 

Signal  filtering  can  be  provided  in  both  the  analog  circuitry  and  software.  This  allows 
checking  for  both  in-band  and  out-of-band  signal  components.  Measurements  can  include 
frequency,  DC  volts,  AC  volts,  peak/valley,  while  computations  can  include  voltage  ratio, 
phase,  etc. 

The  BIT,  as  shown  in  figure  5.4-1,  should  have  the  ability  to  inject  test  signals.  A  remote 
8-channel  signal  selector  integrated  circuit  would  reduce  the  number  of  interconnections 
by  providing  control  with  a  single  wire.  Signal  injection  would  be  through  a  programmable 
series  impedance  or  attenuator. 

Digital  BIT  would  be  able  to  build  upon  the  analog  tools.  It  could  add  an  integrated  circuit 
to  the  chip  set  to  provide  signature  analysis,  transition  counting,  logic  analysis,  and 
circuits  to  look  at  risetime  or  jitter. 

5.4.7  Benefits 

Smart  BIT  and  Smart  System  Integrated  Test  (see  section  5.6)  provide  virtually  the  only 
means  found  by  this  survey  to  reduce  false  alarms  as  well  as  being  able  to  provide  enough 


information  to  the  maintenance  expert  to  reduce  unnecessary  maintenance.  In  addition  to 
the  high  payoff  resulting  from  reduced  maintanance  costs,  Smart  BIT  can  also  reduce 
contractor  costs  by  providing  a  tool  to  detect  and  isolate  faults  during  the  full  scale 
development  phases,  thus  reducing  engineering  effort. 

5.4.S  Developmental  Status 

The  RADC  Smart  BIT  contract  award  (February  84)  to  Grumman  Aerospace  is  the  only 
known  work  in  process..  The  evolutionary  potential  of  Smart  BIT  is  given  in  figure  5.4-2, 
along  with  an  estimated  practical  time  scale  for  the  development. 

The  Artificial  Intelligence  Application  to  Maintenance  Technology  (3)  report  stated  that  a 
complete  series  of  studies  leading  to  the  design  of  an  on-board  knowledge  based 
monitoring  system  or  the  design  or  test  of  experimental  BIT  systems  would  run  2  to  4 
years  and  cost  one  to  six  million  dollars.  Benefits  are  incalculable  since  they  include 
the  worth  of  reduced  mission  aborts  due  to  false  alarms.  More  tangible  benefits  could 
include  a  90%  reduction  of  false  alarms,  and  the  decrease  of  the  portion  of  the  units  sent 
to  repair  which  test  "good"  from  the  present  30%  to  perhaps  10%.  The  report  speculated 
that  successful  application  should  pay  for  itself  in  two  years  of  operation  on  one  system 
and  provide  measurable  improvement  in  the  ready  rate  of  the  system  using  it. 

5.4.9  Risk 

The  development  of  programmable-recording  BIT  /Smart  BIT  is  technically  straight¬ 
forward.  Programmable-recording  BIT  could  be  developed  on  a  practical  basis  now  for 
some  applications.  Smart  BIT  requires  more  development,  but  this  could  be  done  after 
completion  of  the  current  RADC  contract. 


5.5  MAINTENANCE  EXPERT  SYSTEMS 


A  maintenance  expert  system  aids  a  technician  in  performing  functional  and  diagnostic 
tests  on  boxes  or  systems  online  or  offline.  This  is  the  most  advanced  application  of 
artificial  intelligence  to  testability  and  is  well  documented  in  the  literature.  There  are 
several  cost  effective,  generic,  maintenance  expert  systems  that  process  offline  test 
information.  The  four  at  the  top  of  the  table  5.5-1  by  GE,  DEC,  Battelle,  and  ICL  have 
already  been  built  and  tested.  In  addition,  the  GE,  DEC,  and  ICL  systems  saw  factory  use 
in  1983  while  the  much  more  powerful  Battelle  system  was  not  released  to  end  users.  The 
fourteen  other  systems  have  not  yet  been  as  fully  developed. 

The  findings  of  this  section  are: 

o  Cost  effective  maintenance  experts  exist  now 

o  Five  year  evolution  will  include:  self  improving  diagnostics,  prognostics, 
circuit  understanding,  and  discrimination  of  false  alarms 
o  An  engineering  work  station  is  a  practical  device  to  add  maintenance  expert 
capability  to  ATE  and  bench  testing 

o  Testability  improvements  will  further  enhance  the  Maintenance  Expert  per¬ 
formance  and  cost  effectiveness 

o  A  time  critical  Maintenance  Expert  requires  high  accuracy  levels  that  are 
difficult  to  achieve  without  testability  improvements 
o  Maintenance  experts  will  improve  diagnostic  and  functional  test  time  for  ATE 

5.5.1  Maintenance  Expert  Systems  Evaluated 

There  are  five  different  categories  of  maintenance  experts  analyzed  by  this  report  which 
share  many  rules  and  methodologies  and  will  follow  similar  evolutionary  growth  patterns. 
These  range  from  bench  testing  through  system  level  applications.  The  maintenance 
experts  were  split  into  these  five  sections  so  that  evaluations  and  recommendations  could 
be  made  separately  for  each  application,  because  of  the  wide  differences  in  the 
applications.  The  maintenance  experts  are: 

o  Smart  System  Integrated  Test 

o  Maintenance  Expert-Box 

o  Maintenance  Expert-System 


o  Maintenance  Expert-Smart 

(includes  Smart  BIT  and  learning  capability) 
o  Smart  bench 

An  overview  of  the  features  in  these  expert  systems  is  given  in  table  5.5-2. 


FEATURE 

SMART 

SIT 

M.E. 

BOX 

M.E. 

SYSTEM 

M.E. 

SMART  SYSTEM 

SMART 

BENCH 

Unit  tested 

System 

Box 

System 

System 

Box,  Card 

On-line  test 

Yes 

No 

No 

No 

No 

Location  of  testing 

Field 

Depot 

Field- 

Intermediate 

Field- 

Intermediate 

Depot 

Reduce  removals  from  system 

Yes 

No 

Yes 

Yes 

No 

High  accuracy  needed  due  to  time 
critical  task 

Often 

No 

Sometimes 

Sometimes 

No 

Reduce  false  alarms 

Yes 

No 

s 

No 

No 

No 

Smart  -  Optimizes  tests  and  learns 
diagnostics  with  aid  of  smart  BIT 

Yes 

No 

No 

Yes 

Ye* 

Cost  Effectively  applied  to  units  in 
field 

No 

Yes 

No 

No 

Yes 

Table  5.5-2.  Maintenance  Experts  Overview 

The  performance  of  the  expert  systems  is  very  dependent  upon  the  unit's  inherent 
testability  and  ability  to  record  sufficient  data  fault.  Maintenance  experts  will  be  able  to 
evolve  in  three  to  five  years  time  into  SMART  Maintenance  Experts  having  the  ability  to 
learn  by  SID  and  getting  information  from  SMART  BIT.  Initially  the  maintenance  expert 
would  only  have  knowledge  of  a  fault  tree  and  be  able  to  modify  functional  test 
sequences,  but  later  capability  could  be  added  to  automatically  acquire  diagnostic 
information. 


Maintenance  Expert  System*  Approaches  That  Were  Not  Evaluated. 

Research  aimed  at  having  A1  understand  circuit  operation  and  faults  has  been  going  on  at 
Stanford,  MIT  and  Xerox.  This  work  is  summarized  in  table  5.5-1  as  having  high  circuit 
detail  knowledge,  and  all  of  their  developers,  when  surveyed,  agreed  that  at  least  3  to  5 
years  would  be  needed  before  practical  use  is  possible.  Card  level  maintenance  experts 
were  not  included  in  this  study  due  to  lack  of  maturity  of  circuit  understanding  by  expert 
systems  and  since  the  commercial  card  testers  with  ATPG  do  an  acceptable  job. 

Other  special  maintenance  experts  (such  as  Beil  telephone's  ACE  which  uses  AI  to  find  a 
common  source  of  telephone  cable  fault  reports)  were  not  included  because  they  did  not 
appear  to  have  as  widespread  use,  or  solved  needs  which  currently  don't  apply  to  military 
testability. 

Evolution  of  Maintenance  Expert  Systems 

Maintenance  experts,  being  a  subclass  of  rule  based  expert  systems,  can  often  explain 
themselves  when  called  upon,  but  they  have  a  limited  scope  of  knowledge.  Within  that 
scope  of  knowledge,  the  maintenance  experts  act  very  similar  to  the  human  in  terms  of 
being  able  to  process  information  associated  with  functional  and  diagnostic  testing. 
Maintenance  experts  differ  from  rule  based  expert  systems  only  in  scope  of  material.  The 
maintenance  expert's  extremely  limited  scope  of  knowledge  (electronic  testing)  is  now 
allowing  the  creation  of  generic  maintenance  experts  which  can  be  reapplied  to  many 
systems. 

Step  One:  Offline  Maintenance  Expert: 

The  first  stage  of  evolution  (figure  5.5-1)  is  as  an  off-line  aid  that  can  suggest  which  tests 
should  be  made  based  on  previous  tests.  The  Gen-X  system  has  this  capability.  The  DEC 
Intelligent  Diagnostic  tool  (IDT)  is  on-line,  but  does  not  learn  diagnostics  or  functional 
test  optimization.  Current  versions  of  both  might  be  described  as  automated  dependency 
chart  analyzers. 

ARINC's  (Aeronautical  Radio  Inc.)  on-going  development  of  STAMP  (System  Testability 
and  Maintenance  Program-(29))  has  resulted  in  the  optimization  of  test  sequences  based 
on  failure  history  at  a  single  ATE  test  site.  The  Apple  computer  which  is  the  STAMP  host 
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5-1  Maintenance  Expert  Evolution 


does  not  yet  have  any  direct  ATE  connection.  Future  versions  of  STAMP  with  another 
host  computer  may  evolve  into  on-line  use. 

From  the  present  off-line  capability,  the  systems  will  evolve  to  on-line  observation,  and 
then  enhancements  can  be  added  which  utilize  the  power  of  AI  to  control,  interpret,  and 
analyze  the  test  in  progress.  The  anticipated  evolution  from  1983  to  1990  shown  on  figure 
5.5-1  will  be  discussed  in  the  following  paragraphs. 

Step  Two:  On-line  Maintenance  Expert: 

The  next  phase  of  evolution  of  the  maintenance  experts  will  be  the  combination  of  rule 
based  maintenance  experts  and  test  sequence  optimization.  This  will  allow  developers  to 
produce  a  maintenance  expert  that  can  observe  tests  in  progress  at  either  the  system, 
ATE,  or  bench  test  level,  capture  data  and  provide  test  enhancement  form  the  data. 
(Note:  Dr.  Pau's  work  may  have  already  progressed  to  this  stage.  His  modular  system  has 
demonstrated  combining  of  vision  information  and  card  test  information). 

CRIB,  a  Computer  Retrieval  Incidence  Bank  expert  system  (10),  is  used  for  computer 
hardware  and  software  fault  diagnosis.  Knowledge  is  added  to  it  by  maintenance 
personnel  giving  it  self  improving  diagnostics  from  an  aggregate  of  users.  Its  daily  use 
provides  its  validation.  Knowledge  from  previous  test  results  provides  groups  of 
symptoms  which  can  uniquely  identify  a  fault.  It  is  not  necessary  to  have  a  complete 
match  of  symptoms.  CRIB  is  based  on  a  small,  fixed  set  of  rules  and  me ta -rules  and  a 
large,  changing  body  of  data.  This  MYCIN-like  maintenance  expert  system  (16)  has  been 
in  use  at  International  Computers  Limited  for  several  years,  and  as  such,  is  the  most 
mature  system  reported  in  the  literature. 

Modular  Maintenance  Expert 

A  maintenance  expert  can  be  used  with  many  different  environments,  different  ATE  and 
with  bench  testing  as  shown  in  figure  5.5-2.  The  connection  to  the  ATE  for  observation 
can  be  as  simple  as  monitoring  the  line  wh  'ch  connects  the  ATE  with  its  system  console, 
generally  an  RS-232  interface.  Most  of  tht  ormation  concerning  test  step  number  and 
results  of  the  tests  is  available  on  this  line,  in  some  cases  it  may  be  necessary  to  have  an 
additional  connection  or  to  monitor  the  ATE  line  printer.  This  modular  maintenance 
expert  approach  has  the  capability  of  doing  all  of  the  information  processing  mentioned  in 


the  introduction  to  this  section,  including  acquiring  test  information  from  the  ATE, 
storing  it,  analyzing  it  locally,  and  distributing  this  information  back  to  the  central  site 
for  incorporation  in  diagnostics.  Engineering  work  stations  have  the  capability  to  provide 
a  common  connection,  without  modification,  to  many  military  ATE. 
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Figure  5.5-2.  Modular  Maintenance  Experts. 


Step  Three  Modular  Maintenance  Expert  Enhancements 

The  next  phase  of  the  maintenance  expert's  evolution  involves  enhancements  to  the 
modular  maintenance  expert's  capability.  These  five  enhancements  are  essentially 
independent,  and  can  be  implemented  in  any  combination  or  sequence. 


1.  Integrated  Display 

The  modular  maintenance  expert  will  be  able  to  provide  an  integrated  information  display 
on  its  graphics  screen.  This  will  provide  the  test  technician  with  much  additional, 
relevant  information.  Data  concerning  the  current  and  past  results  of  the  unit  under  test 
can  be  superimposed  upon  the  nodal  connection  diagram  of  the  unit.  Color  would  provide 
the  best  display.  Even  though  color  is  not  widely  available  on  engineering  work  stations 
today,  it  should  be  readily  available  by  the  time  this  step  is  implemented.  The  graphic 


display  provides  the  technician  an  integrated  view  of  test  results  from  which  he  can 
perform  diagnostic  isolation  more  easily. 

2.  Control  Test  Sequence 

Another  enhancement  will  be  the  ability  to  control  the  test  sequence.  Currently,  STAMP 
knows  how  to  control  the  test  -sequence,  but  is  unable  to  do  so  directly.  With  today's 
software  configuration  control  requirements,  it  is  doubtful  that  the  military  environment 
will  permit  the  arbitrary  changing  of  test  sequences  upon  the  decision  of  a  software 
maintenance  expert.  Some  of  the  maintenance  expert  rules  will,  however,  enable  it  to 
perform  what  would  appear  to  be  an  arbitrary  resequencing.  The  rules  will  dictate  which 
parts  of  the  tests  must  be  executed  before  others,  e.g.,  initialization.  These  sequence 
rules  will  enable  the  maintenance  expert  to  control  the  test-in-progress  during  diagnos¬ 
tics,  which  will  be  especially  useful  for  isolation  of  intermittents. 

3.  Analyze  Test  Results  Near  Limits 

The  maintenance  expert  can  have  rules  to  analyze  test  results  which  are  close  to  limits  of 
acceptability.  The  "fuzzy"  information  that  a  test  result  is  near  a  limit  is  very  important 
for  the  diagnosis  of  marginal  equipment  and  intermittents.  This  information  is  also  useful 
for  prognostics  (fault  isolation  before  a  hard  failure  has  occurred).  The  maintenance 
expert  can  use  its  previous  diagnostic  experience  to  determine  the  significance  of  a  result 
near  a  limit.  If  the  event  is  significant,  it  can  be  displayed  graphically  on  the  screen  with 
color,  flashing,  shading,  etc.  Such  information  should  prove  invaluable  for  diagnosing 
problems  that  today  typically  result  in  RTOK  (ReTest  OK)  and  CND  (CanNot  Duplicate) 
situations. 

4.  Analyze  Signal  Quality  Change 

The  maintenance  expert  which  is  in  control  of  the  test  sequence  can  sometimes  enhance 
the  test  by  looking  at  signal  quality.  This  is  most  easily  implemented  via  the  SMART 
bench  approach  described  there.  The  ability  to  monitor  signal  quality  can  be  used  by  all 
of  the  maintenance  experts. 

5.  After  Alarm,  Determine  when  Unit  is  Again  Operating  Reliably 

The  SMART  system  integrated  test  can  be  enhanced  to  develop  rules  to  determine  when  a 
unit  is  again  working  reliably  after  an  alarm,  thus  reducing  removals  and  providing 
recovery  capability  after  soft  faults. 
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Testability  Improvement 


To  make  full  use  of  the  capability  of  the  enhanced  maintenance  experts,  the  unit  under 
test  must  have  improved  built-in  testability  over  that  which  is  typically  designed  in  now. 
Without  improved  testability,  the  maintenance  expert  will  only  be  able  to  produce 
statistical  diagnostics  e.g.,  70%  of  failures  of  a  node  are  due  to  component  A,  20%  due  to 
component  B,  etc.. 

It  should  also  be  noted  that  when  future  improvement  in  testability  reduces  diagnostic 
ambiguity,  there  will  be  a  reduced  need  for  self-improving  diagnostics. 

Benefits 

Maintenance  experts  can  substantially  reduce  the  cost  of  testing  even  without  direct  test 
observation  or  learning  of  diagnostics.  Future  functional  and  diagnostic  testing  based  on 
AI  rules  reduces  costs  bys 

o  Inclusion  of  diagnostics  automatically  with  decreased  need  for  re-verification 
and  validation  of  test  programs 

o  Reduction  in  test  time  due  to  real  modification  of  test  sequences 
o  Reducing  ATE  obsolescence  by  use  of  the  modular  maintenance  expert 
o  Allowing  standardization  of  operator  interfaces  and  reporting  mechanisms 
o  Reducing  the  need  for  using  the  bench  for  diagnosis  of  fault  finding  when  the 
ATE  cannot  be  cost  effectively  used 

Battelle:  Dr.  Pau's  Work  is  Best  Surveyed 

Dr.  Pau’s  work  at  Battelle  in  Switzerland,  is  clearly  far  ahead  of  all  other  maintenance 
experts  surveyed  by  this  contract.  Under  internal  R&D  funding,  Dr.  Pau  has  developed 
400  metarules  for  domain  independent  diagnostics  (11).  These  metarules  fall  into  three 
classes: 

o  Diagnostics  through  verification 

o  Testing  while  stressing 

o  Rules  to  detect  errors  of  design  and  interface  types 


Metarules  can  be  thoroughly  tested  and  then  re-used  on  subsequent  systems,  greatly 
reducing  the  cost  for  producing  a  maintenance  expert.  Dr.  Pau  has  extensively  published 
in  this  area  (see  8.2  -  Maintenance  Expert)  and  his  1981  book  (15)  is  the  only  authoritative 
treatment  of  sophisticated  diagnosis  without  the  use  of  artificial  intelligence.  Dr.  Pau's 
work  is  shown  in  the  evolutionary  chart  of  maintenance  experts  (fig.  5.5-1)  as  the  most 
sophisticated  artificial  intelligence  diagnostics  system  in  current  use. 

Tangible  Tools 

The  experience  of  maintenance  expert  development  at  G.E.  (8),  DEC  (9),  ICL  (10),  and 
with  Dr.  Pau  has  shown  that  it  is  possible  to  generate  generic  maintenance  experts  which 
can  be  inexpensively  applied  to  other  systems.  Similarly  it  appears  that  the  hardware  of 
the  modular  maintenance  expert  can  be  developed  and  used  in  many  test  situations.  The 
modular  maintenance  expert  should  be  usable  with  very  little  modification  for  many  of 
the  ATE  currently  used  by  the  military  as  well  as  in  new  ATE  and  SMART  bench 
applications. 

Maintenance  Expert  Time  Criticality  Determines  Required  Accuracy 

A  system  which  requires  high  availability  places  high  demands  for  accuracy  on  a 
maintenance  expert  system.  For  factory  or  depot  testing,  a  maintenance  expert  which 
isolates  faults  to  a  single  unit  70%  of  the  time  is  a  useful  aid.  This  level  of  ambiguity  is 
not  acceptable,  however,  for  systems  where  availability  is  optimized  and  a  quick  test 
after  each  replacement  cannot  be  performed.  Both  commercial  and  military  aircraft,  for 
example,  require  good  assurance  that  the  fault  is  corrected  to  reduce  aborted  flights. 
Figure  5.5-3  indicates  that  time  critical  diagnostics  must  isolate  more  than  90%  of  faults 
in  a  system  so  that  the  maintenance  technician  will  not  remove  all  items  in  the  ambiguity 
group.  This  increased  accuracy  will  result  from  improved  inherent  testability  and  not 
from  "self  improving  diagnostics". 

Figure  5.5-3  is  based  on  conversations  with  technicians  from  both  time  critical  and  non¬ 
time  critical  maintenance  organizations  testability  requirements  specifications,  and 
Planetalk  magazine  which  reviews  maintenance  procedures  for  commercial  aircraft. 


Figure  5.5-3.  Maintenance  Expert  Use  Dependent  on  Time  Criticality 

Military  Maintenance  Expert  Verification/Validation 

There  is  no  evidence  of  a  current  verified  and  validated  rule  based  system  used  for  the 
military.  When  rules  are  generated  by  an  expert  for  a  specific  system  there  is  a  concern 
about  the  correctness  and  completeness  of  the  rules.  The  use  of  system  independent  rules 
and/or  machine  learning  from  field  data  will  substantially  decrease  the  effort  needed  to 
generate  each  maintenance  expert  system.  Special  issues  of  concern  are  time  varying  and 
noisy  data.  The  first  two  articles  in  8.2  -  AI  MISC.  deal  with  the  role  of  time  in  AI. 
Noisy  information  is  covered  by  books  such  as  Fuzzy  Techniques  in  Pattern  Recognition  by 
A.  Kandel.  At  some  point  in  the  evolution  of  military  maintenance  expert  systems  the 
effects  of  time  and  noisy  data  will  have  to  be  considered. 
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5.6  SMART  SYSTEM  INTEGRATED  TEST 


The  Smart  System  Integrated  Test  (Smart  SIT)  is  very  similar  to  Smart  BIT,  except  that  it 
applies  to  an  entire  system,  rather  than  to  just  a  single  box  of  electronics.  By  intelligent 
data  recording  it  will  reduce  fault  isolation  ambiguity  and  false  alarms,  especially  for 
intermittents,  either  directly  or  indirectly. 


The  Integrated  Test  and  Maintenance  (1)  contract  by  Boeing  studied  the  use  of  AI  for 
improved  system  level  testing.  While  being  similar  to  the  Smart  BIT  concepts,  there  is 
generally  less  time  for  data  analysis  in  the  fighter  aircraft  environment  which  was  the 
focus  of  the  ITM  contract  (table  5.6-1).  The  use  of  fault  data  recording  with 
environmental  related  information  was  essential  to  post-flight  analysis  that  used  AI 
techniques  to  determine  fault  cause  (see  8.4.3).  Smart  SIT  can  provide  interactive 
surveillance  of  system  operation  using  generic  rules  for  relevant  patterns  and  attributes. 


Establish  operational  readiness 

Continual  fault  monitoring 

Fault  detection  (X%) 

Preflioht 

E 

E 

Infliqht 

E 

E 

Postfliqht 

E 

Support  reconfiguration 

Minimal  pilot  interface 

E 

E 

E 

Fault  isolation  to  support  reconfinuration 

E 

E 

Fault  isolation  to  LRU  (X%) 

Fault  isolation  to  SRU  (X“/i) 

D(2) 

DO) 

E 

D(3) 

Maintain  system  status 

E 

E 

Fault  data  recording 

Fault  record  processing 

Report  generation 

E 

E 

E 

E 

D(7) 

Manual  test  interaction 

Manual  isolation 

Manual  bit  interrogation 

E 

E 

E 

E 

Automated  tech  orders 

D(5) 

D(6) 

No  additional  equipment 

E 

E 

0(4) 

Aircraft  power  only 

E 

E 

E 

Intermittent  handling 

E 

E 

E 

False  alarm  requirement 

Environmental  data  recording 

Multiple  fault  isolation 

E 

D(8) 

E 

0(9) 

E  =  Essential 

P(n)  =  Desirable  (rank) 


Table  5.6-1.  Smart  System  Integrated  Test— Essential  or  Desirable  Capabilities 


5.7  COMPARISON  OF  BOX  AND  SYSTEM  MAINTENANCE  EXPERTS 


Section  5.5  referred  to  many  box  and  system  level  maintenance  experts  and  they  are 
similar  in  that  box  and  system  level  both: 
o  Perform  offline  tests 

o  Have  no  access  to  Smart  BIT 

o  Cannot  learn 

However,  system  level  experts  have: 
o  More  functions  to  monitor 

o  More  items  to  isolate  a  fault  to 

o  More  organizations  to  obtain  test  information  from 

o  Better  functional  partitioning  (since  separate  organizations  created  boxes) 
o  Time  critical  decisions  (box  removal) 

It  may  be  possible  to  generate  a  system  level  maintenance  expert  for  a  similar  cost  as  for 
a  box.  The  fact  that  the  system  level  can  reduce  false  removals  gives  it  much  more 
benefit  than  the  box  maintenance  expert.  However,  as  is  noted  in  the  evaluation  section,, 
a  system  level  maintenance  expert  requiring  high  accuracy  is  too  risky  in  the  near  term. 
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M  SMART  BENCH  TESTING 


Brief  Description 

The  Smart  Bench  concept  developed  by  Boeing  is  a  method  to  semiautomate  bench  testing 
by  use  of  artificial  intelligence,  which  provides  a  substitute  for  manual  testing.  Smart 
Bench  uses  a  computer  to  perform  bench  tests  of  electronics.  It  has  the  capability  of 
learning  both  functional  and  diagnostic  information.  It  also  would  use  the  standard 
instrument  interface,  the  IEEE  488  bus,  which  enables  the  computer  to  control  some  of 
the  bench  test  equipment  (fig.  5.8-1).  Other  equipment  would  remain  manually  operated. 
The  computer  would  first  ''learn”  the  functional  tests  that  are  performed  by  the  human 
operator  with  the  test  equipment.  Subsequent  functional  tests  would  be  performed 
semiautomatically,  with  the  computer  verifying  that  the  test  stimuli  had  been  applied, 
whether  by  the  human  or  the  computer. 

Use  of  a  multiplexer  switch  would  enable  the  computer  to  measure  any  two  lines  of  the 
unit  under  test,  thus  allowing  confirmation  of  stimulus  and  measurement  of  response.  The 
stimulus  can  be  applied  to  the  unit  under  test  by  use  of  patch  cords  or  already  existing 
Special  Test  Equipment  (STE). 

The  next  stage  of  the  Smart  Bench  development  would  include  application  of  incorpora¬ 
tion  of  diagnostic  experience  through  automated  learning  (IDEAL)  previously  discussed  in 
section  4.1. 

Smart  Bench  Can  Test  Units  Already  in  Field 

Smart  Bench  could  be  used  to  test  either  previously  fielded  units  or  those  of  new  design. 
This  is  one  of  the  few  AI  testability  applications  which  can  be  applied  to  fielded  units.  It 
should  be  especially  useful  for  troublesome  units  or  those  where  little  diagnostic  guidance 
has  been  provided.  In  addition,  Smart  Bench  can  be  used  for  factory  production  testing 
and  to  aid  in  the  unit  design  and  validation. 

Smart  Bench  Can  Test  Some  Units  Which  Might  Otherwise  Be  On  ATE 

Smart  Bench  can  semiautomate  bench  testing,  and  can  also  be  used  to  test  units  where 
ATE  is  not  cost  effective.  Most  military  testing  is  performed  using  ATE,  whether  ATE  or 


Figure  5.8-1  Smart  Bench  Diagram 


bench  testing  is  used  depends  on  such  factors  as  number  of  units  to  be  tested  and  the 
length  and  complexity  of  test.  For  those  units  on  the  border  between  ATE  and  bench, 
Smart  Bench  can  provide  a  viable  new  alternative. 

Benefits:  Cost  Effective  Development  and  Execution  of  Tests 

Smart  Bench  semiautomatic  testing  can  be  developed  more  cost  effectively  than  ATE  or 
manual  bench  testing  and  can  execute  functional  tests  more  effectively  than  manual 
testing.  In  addition,  the  capability  of  Smart  Bench  to  incorporate  diagnostic  experience 
will  materially  decrease  the  bench  maintenance  time  for  all  technicians  (see  figure  4.1-1). 

For  hardware  development  programs  of  low  complexity  or  small  volume,  it  may  be 
possible  to  use  Smart  Bench  testing  for  all  units. 

Smart  Bench  Provides  AI  Application  Test  Bed 

Smart  Bench  can  provide  a  test  environment  for  several  of  the  concepts  needed  by  the 
other  AI  applications,  including  Maintenance  Experts  and  Smart  BIT.  The  Smart  Bench 
environment  will  provide  a  low  cost  means  to  develop  the  following  five  concepts: 
o  IDEAL 

o  Ability  to  modify  fault  tree 

o  Graphical  display 

o  Man- machine  interactor 

o  Semiautomatic  testing 

IDEAL  (incorporation  of  Diagnostic  Experience  through  Automated  Learning) 

IDEAL  can  be  implemented  by  several  different  AI  and  non-AI  techniques.  Using  Smart 
Bench  as  a  testbed  for  the  AI  applications  will  result  in  less  development  risk  for  the 
larger  AI  applications  of  smart  system  integrated  test  and  maintenance  experts  than 
development  of  IDEAL  using  ATE.  The  Smart  Bench  evolution  figure  (5.8-2)  projects 
IDEAL  as  being  incorporated  in  the  second  phase  of  development. 
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Figure  5.8-2  Smart  Bench  Evolution 


Ability  to  modify  fault  trees  (test  sequence  optimized  by  observed  failures) 

Smart  Bench  could  learn  to  spot  the  degradations  in  signal  quality,  in  the  same  way  they 
would  be  noticed  by  technicians  familiar  with  the  unit.  The  signal  quality  information 
obtained  by  IDEAL  would  be  invaluable  in  diagnosing  intermittent  units  and  in  reducing 
the  unit  specific  experience  needed  by  the  test  technician,  thereby  reducing  training  and 
increasing  efficiency. 

Integration  of  information  into  a  graphical  display 

Smart  Bench  can  provide  integrated  information  graphically  to  the  technician.  The 
information  can  come  from  the  circuit  diagram  or  via  the  technical  order  incorporating 
stimulus  and  response  data.  The  status  of  the  unit  under  test  can  be  shown  graphically 
along  with  the  circuit  diagram  or  functional  diagram  to  indicate  if  the  test  was  O.K., 
marginal,  failed  high  or  low,  intermittent,  if  the  signal  was  of  poor  quality,  and  if  a  fault 
was  detected,  where  it  was  isolated. 

Man-machine  interaction.  (Natural  Language,  etc.) 

Use  of  voice  to  improve  the  man/machine  interface  capture  of  information  during  manual 
probing  will  be  a  useful  future  (5  year)  productivity  enhancement.  With  voice  recognition, 
the  technician  would  not  have  to  record  diagnostic  experience  when  his  hands  are  busy 
with  manual  probing. 

The  technology  now  exists  for  reading  numerical  displays;  thus,  Smart  Bench  could  be 
enhanced  with  a  TV  camera  and  pattern  recognition  capabilities  to  read  displays  of  units 
under  test.  The  displays  to  be  read  could  include  numeric  LED  readouts,  CRT  data 
indicator  lights,  etc.  (see  section  2.7). 

Semiautomatic  Testing 

With  Smart  Bench,  many  response  measurements  will  be  made  without  operator  interven¬ 
tion.  This  will  result  in  more  consistent  and  improved  testing  with  fewer  units  damaged 
by  errors  in  manual  testing.  The  technician  will  also  have  more  information  for 
understanding  the  diagnostics.  This  should  result  in  reduced  frustration  and  increased 
employee  longevity,  thereby  reducing  training  costs. 


Smart  Bend)  Can  Be  Used  By  Many  Projects  Without  Modification 


The  capability  of  Smart  Bench  at  the  end  of  the  second  stage  of  evolution  can  be  used  by 
engineers  and  technicians  for  developing  their  own  Smart  Bench  tests  with  existing  bench 
test  equipment.  This  tool  would  consist  of  the  Engineering  Work  Station  software  and  the 
multiplexer  switch  shown  in  the  Smart  Bench  block  diagram  (fig.  5.S-1).  By  eliminating 
the  ATE  matrix  switching  capability,  and  by  only  having  the  ability  to  monitor  two 
channels  of  responses  simultaneously,  the  cost  of  the  Smart  Bench  system  is  only  one 
tenth  to  one  hundredth  that  of  an  ATE  system.  This  is  offset  by  the  increased  personnel 
costs  due  to  slower  tests. 

Development  Status 

Smart  Bench  is  a  concept  developed  by  Boeing  that  has  not  yet  been  implemented.  While 
not  producing  overall  large  cost  savings,  it  can  provide  a  means  to  prototype  maintenance 
expert  concepts. 

Risk 

Smart  Bench,  as  proposed,  uses  established  A1  techniques  that  have  been  proven  in  other 
applications.  For  that  reason  there  is  little  risk  in  the  application  of  machine  learning 
and  rule  based  expert  systems  to  this  bench  environment. 
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5.9  AUTOMATIC  TEST  PROGRAM  GENERATION  (ATPG) 


Summary 

o  AI  is  being  applied  to  Digital  ATPG 

o  The  opportunity  for  savings  is  low  due  to  a  segmented  market 

ATPG  is  used  to  create  ATE  functional  tests  for  digital,  analog  and  computer  based  cards 
and  boxes.  It  uses  simulation  capability  on  a  large  computer  (often  a  VAX)  to 
automatically  generate  these  electrical  tests  and  is  frequently  used  for  testing  of  digital 
elecronics  for  both  commercial  and  military  electronics  (see  book,  Bennetts,  "Introduction 
to  Digital  Board  Testing",  19S2). 

Benefits  of  AI 

Robinson  (33)  states  that  AI  would: 

o  Produce  well  structured  tests 

o  Produce  tests  more  quickly  by  using  hierarchical  functional  knowledge 
o  Be  extensible 

A  block  diagram  of  Robinson's  test  generator  is  shown  in  figure  5.9-1.  A  very  large  ATE 
manufacture  (Gen  Rad)  is  funding  the  work  primarily  for  digital  electronics.  It  is  doubtful 
that  much  improvement  in  coverage  of  digital  functional  tests  is  expected,  as  that  is 
limited  by  the  amount  of  DFT  (fig.  3.2-2). 

Analog  ATPG  (A ATPG)  with  AI  is  the  focus  of  a  5  year  Navy  RJcD  plan  for  the  mid  1980's. 
There  has  been  very  little  success  with  analog  ATPG  without  AI,  due  in  part  to  a 
defocussed  and  decreasing  market.  The  over  40  dialects  of  the  ATLAS  language  required 
for  military  analog  ATE  tests  coupled  with  lack  of  standard  ATE  and  decreasing  use  of 
analog  electronics  makes  for  a  small  market  for  an  AATPG. 

The  main  impact  of  AI  on  ATPG  will  probably  be  the  insertion  of  functional  circuit 
knowledge  into  the  simulation.  Presently  both  analog  and  digital  circuits  are  simulated  at 
a  very  low  level  that  does  not  include  knowledge  of  the  circuit  function.  Tests  are  not 
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well  structured  and  cannot  be  quickly  simulated  if  they  try  to  test  all  possible  functions  of 
a  circuit  rather  than  testing  the  function  designed  for. 

Limited  Opportunity  for  Cost  Savings 


When  compared  with  the  other  AI  opportunities  evaluted,  ATPG  is  not  seen  as  having  as 
many  benefits.  It  does  not: 

o  Improve  testability 

o  Decrease  test  false  alarms,  etc. 

o  Decrease  test  time  significantly 

Self  Test  Capability  Decreases  Need  for  ATPG 


The  increasing  capability  of  new  units  under  test  should  permit  increasing  use  of  BIT. 
Functional  BIT  capability  in  excess  of  95%  is  being  required  of  new  military  electronics, 
which  should  decrease  the  ATE  ATPG  requirements.  Increasing  use  of  VLSI  with  self-test 
capability  will  have  a  similar  effect  on  ATPG. 
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Figure  5.9-1  HITEST  Test  Generator  -  Block  Diagram 


5.10  REQUIREMENTS  FOR  DEVELOPMENT  AND  USE  OF  AI  APPLICATIONS 


A  typical  government  research  and  development  effort  will  not  result  in  a  product  that 
can  be  directly  used  by  contractors.  To  maximize  the  cost  effectiveness  of  the  AI 
opportunities  it  is  desirable  to  have  all  developers  and  end  users  of  the  applications  make 
use  of  the  same  hardware  and  software.  This  would  permit  a  synergistic  sharing  of 
development  techniques  and  application  information. 

Industry  standardization  of  engineering  work  stations  is  providing  a  breakthrough  in  low 
cost  development  and  widespread  use  of  AI  opportunities.  Compatible  machines  will  soon 
be  directly  available  to  designers  and  developers. 

Developers  Need  AI  Language  and  Graphics 

The  developers  of  the  AI  techniques  need  a  machine  that  can  execute  an  AI  language  such 
as  LISP  or  PROLOG.  They  also  need  a  high  resolution  graphics  display  for  their  own 
program  development  and  as  a  means  to  display  diagrammatic  maintenance  information 
such  as  a  circuit.  Often,  when  the  developers  have  optimized  the  machine  for  their  own 
needs  they  have  selected  machines  designed  to  directly  execute  an  AI  language  such  as 
models  by  Xerox,  LMI,  or  Symbolics.  While  this  does  offer  faster  execution  of  an  AI 
language  to  ease  development  tasks  the  AI  language  still  does  not  execute  fast  enough  for 
a  majority  of  AI  applications.  Because  of  that,  it  must  later  be  re-coded  in  a  procedural 
language  (typically  Pascal),  and  re-hosted  on  another  machine. 

Product-Designers  Need  Industry  Standards 

The  product  designers  among  contractors  and  subcontractors  want  tools  that  can  be  used 
without  having  to  buy  new  equipment  or  having  to  rehost  the  tools  to  their  own 
equipment,  which  results  in  a  long  term  software  maintenance  burden.  Industry  standards 
result  in  multiple  sources  of  compatible  hardware  and  software.  Sole  source  systems, 
such  as  the  LISP  language  AI  machines,  are  not  cost  effective  as  corporate  standards  for 
long  term  operation,  so  they  will  not  be  widely  available  in  a  company. 

There  are  currently  three  standard  low  to  medium  cost  computers  widely  used  by  the 
aerospace  contractors:  the  DEC,  VAX  family  (medium  cost),  IBM  PC  (very  low  cost),  and 
the  emerging  low  cost  engineering  work  stations,  (with  the  Apollo  and  Daisy  based 


systems  having  the  majority  of  the  market,  Electronics,  May  17,  1984).  Several  expert 
systems  have  been  developed,  notably  by  General  Electric  (8)  and  D.  R.  Michalski  (18). 
These  were  developed  on  a  VAX  and  then  executed  on  an  IBM  PC;  however,  both  of  these 
computers  lack  the  good,  standard,  high  resolution  graphics  which  are  of  great  value  in  a 
design  or  maintenance  application.  The  engineering  work  station  is  able  to  supply  the 
power  of  a  VAX  processor  to  a  single  user  and  include  standard  higher  resolution  graphics. 
The  engineering  work  station  is  now  an  excellent  host  for  testability  expert  systems. 
Table  5.10-1  provides  a  comparison  of  the  machine  approaches  just  discussed  in  terms  of 
whether  the  machine  meets  AI  application  to  testability  development  and  application 
needs. 


SPEED 

CAPACITY 

GRAPHICS 

USED  BY 

DESIGNERS 

LISP  Machines 

Yes 

Yes 

Yes 

No 

IBM  PC 

No 

No 

OK 

Yes 

VAX 

Yes 

Yes 

No  Standard 

Yes 

Eng.  Work  Station 

OK 

Yes 

Yes 

Yes 

Table  5.10-1  Computer  Attributes 
Depot  Maintenance  Needs  Connection  to  ATE 

Maintenance  expert  system  applications  described  in  this  report  initially  need  to  monitor 
ATE  operation,  but  will  evolve  into  controlling  ATE  operation.  While  it  is  possible  to 
conceive  of  designing  new  ATE  with  integral  AI  capability,  it  is  much  more  feasible  to 
have  the  AI  function  added  to  the  front  of  an  existing  ATE. 

The  Smart  Bench  and  the  ATE  application  both  need  a  small  modular  computer  with 
circuit  knowledge,  good  graphics,  and  a  means  to  record  diagnostic  information. 

Engineering  Work  Station  Provides  Cost  Effective  Testability 

The  engineering  work  station  provides  a  breakthrough  in  cost  effectiveness  by  being  able 
to  provide  a  design  tool  with  capability  that  exceeds  that  available  on  mainframes  just  a 
few  years  ago.  It  provides  a  single  host  which  satisfies  the  needs  of  developers,  product 
developers,  and  end  users.  As  shown  in  table  5.10-2,  it  also  provides  the  AI  language 
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needed  during  initial  development,  and  the  procedural  language  needed  for  high  speed 
execution,  as  well  as  circuit  modeling  capability,  high  resolution  graphics,  etc. 
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Table  5.10-2 

Engineering  Work  Station  (EWS)  Will  Meet  Testability  Application  Requirements  by  1985 

The  technological  progress  made  by  semiconductor  technology  has  provided  in  the 
engineering  work  station  an  interactive  system  which  is  big  enough,  fast  enough  and  cheap 
enough  to  be  really  useful  in  a  wide  range  of  applications.  This  single  user  computer  does 
not  need  to  communicate  with  any  other  machine  to  perform  its  tasks,  and  gives  the 
designer  more  net  productivity  than  he  usually  has  with  the  shared  VAX  computer.  There 
are  overviews  of  engineering  work  stations  in  the  April  84  issue  of  IEEE  Computer 
Graphics  and  Applications,  which  includes  a  description  of  the  Apollo  Domain  system. 
The  same  magazine  in  Dec  1983  describes  the  Mentor  Graphics  system  in  an  article 
entitled,  "Tools  for  Computer-Aided  Engineering,"  by  T.  Bruggere.  Further  work  station 
references  are  in  the  CAD  part  of  section  8.2. 
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It  is  interesting  to  note  that  while  semiconductor  technology  has  increased  unit  complex¬ 
ity  and  thus  been  the  cause  of  testability  problems,  semiconductor  technology  has  also 
made  it  possible  for  engineering  work  stations  to  be  able  to  cure  the  testability  problems 
(see  figure  5.10-1). 

Standard  Hosts  Can  Be  Used  Throughout  Most  Applications 

Standard  computers  and  standard  knowledge  bases  can  be  used  throughout  all  phases  of  all 
the  applications  evaluated.  While  the  AI  techniques  can  be  put  on  an  assortment  of 
general  purpose  computers  or  special  purpose  LISP  machines,  the  sharing  of  development 
results  among  the  applications  is  the  most  important  factor  and  will  produce  large  savings 
in  time  and  money. 

In  each  of  the  evaluated  opportunities  there  are  three  types  of  groups  involved: 

1)  Research  and  Development 

2)  Product  Designers  who  configure  product  to  their  project 

3)  Product  users 

Substantial  savings  can  be  realized  whenever  a  single  application  can  use  standard 
hardware  and  software  for  all  three  groups.  The  cost  of  rehosting  an  AI  application  for 
each  user  (subcontractor)  would  otherwise  inhibit  its  extensive  use.  It  should  be  noted 
that  the  cost  for  some  engineering  work  stations  will  drop  below  $10,000  in  1984.  The  low 
hardware  cost  should  be  an  incentive  for  standardization. 

As  of  1983,  in  excess  of  50  manyears  of  work  has  gone  into  software  tools  to  be  used  on 
engineering  work  stations  like  the  Mentor  Graphics  systems,  which  are  needed  by  AI 
applications  to  testability.  It  takes  many  manyears  of  work  to  develop  real  circuit 
modeling  tools,  much  less  the  hierarchical  interrelationships  which  are  needed  for  more 
complex  designs.  The  LISP  machines  offer  virtually  no  support  of  electronics  concepts. 
The  AI  application  candidates  being  evaluated  by  this  study  were  each  examined  on  their 
potential  use  of  engineering  work  stations;  the  results  are  presented  in  table  5.10-3. 
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Table  5.10-3  Engineering  Work  Stations  (EWS)  Used  by  All  Applications  in  Many  Phases 


APPLICATION  _ PHASE 

CONFIGURE  PRODUCT 


R&D 

TO  PROJECT 

USER 

CAPDT 

EWS 

EWS 

EWS 

SMART  Bit 

EWS 

EWS 

EWS  used  to  analyze  and 

program  it 

Maint.  Expert-BOX 

EWS 

EWS 

EWS  controls  ATE 

SMART  SIT 

EWS 

EWS 

Uses  militarized  CPU 

SMART  Bench 

EWS 

EWS 

EWS 

ATPG 

EWS 

EWS 

EWS 

Maintenance  Experts  Can  Use  Functional  Circuit  Information  On  Work  Station 

While  the  maintenance  expert  systems  analyzed  by  this  report  do  not  claim  any  knowledge 
of  circuit  function,  and  so  cannot  be  used  for  card  level  diagnosis,  a  future  marriage  of  AI 
circuit  descriptions  with  the  engineering  work  stations  may  provide  this  capability.  The 
engineering  work  station  could  also  interactively  simulate  possible  faults  in  a  circuit  to 
aid  in  diagnosis  of  faults  other  than  the  single,  stuck  at  "0"  or  "1"  type  of  fault  usually 
modeled.  Other  non-AI  tools  like  reliability  and  maintainability  analysis  will  probably  also 
be  added  to  the  CAPDT  on  the  engineering  work  station,  as  they  have  many  of  the  same 
concerns  and  data. 


6.0  EVALUATION  OF  OPPORTUNITIES 


Summary 

The  evaluation  of  the  AI  opportunities  is  presented  in  two  stages.  In  the  first  stage,  an 
evaluation  matrix  details  th$  life  cycle  costs  or  savings  at  each  phase  of  development  for 
the  application  opportunities  identified  in  Section  5.0.  Because  a  system  level  application 
results  in  much  higher  life  cycle  cost  leverage,  the  cost  evaluation  uses  a  weighting  for 
systems  level  applications  several  times  that  of  box  or  card  level  opportunities;  a  factor 
of  ten  was  chosen  as  a  first  approximation  based  on  the  potential  cost  leverage  of  an 
improvement  to  a  system  over  a  box  or  card.  AI  opportunities  were  ranked  on  the  basis 
of  estimated  life  cycle  cost  reduction  provided,  from  greatest  to  leasts 

%  Reduction 


Maintenance  Expert  System  -  Smart  6.4 

Smart  System  Integrated  Test  6.3 

Computer  Aided  Preliminary  Design  for  Testability  6.2 

Maintenance  Expert  -  System  3.7 

Smart  Bit  0.6 

Smart  Bench  0.4 

Maintenance  Expert  -  Box  0.4 

Automatic  Test  Program  Generation  -  Analog  0.2 

Digital  Automatic  Test  Program  Generation  0.2 


Weighted  values  based  on  the  following  criteria  were  then  added  to  the  life  cycle  cost 
evaluation  to  arrive  at  a  figure  of  merit  for  each  application: 

o  Risk  of  Demonstration  in  3  Years 

o  Development  Status 

o  Cost  to  Obtain  Demonstrated  Product 

o  Cost  to  Develop  Tool 

o  Usefulness  of  Resulting  Tool 

o  Retrofit  Capability 

o  Payoff  Leverage  (Benefit/Cost) 

o  Mission  Critical  Function 

o  Enabling  Technology 


The  figures  of  merit  were  separated  into  three  categories  ("1"  being  high),  with  at  least  a 
4  point  gap  between  the  high,  medium,  and  low  recommendations 


Category  _ Application _ 

1  Computer  Aided  Preliminary  Design  for  Testability 

2  Maintenance  Expert  -  System 
Smart  Bit 

Smart  System  Integrated  Test 
Maintenance  Expert  -  Smart 
Maintenance  Expert  -  Box 

3  Smart  Bench 

Digital  Automatic  Program  Gen.  with  Machine  Learning 
Automatic  Test  Program  Generation  -  Analog 

6.1  LIFE  CYCLE  COST  EVALUATION 


This  section  uses  a  matrix  format  to  evaluate  die  life  cycle  costs  and  benefits  for  each 
application  opportunity  separately.  Seven  evaluation  criteria  were  used  and  each  AI 
application  evaluated  for  impact  on  the  seven  phases  of  a  product's  life  cycle.  Discussion 
of  the  terms  used  in  the  evaluation  matrix  are  presented  first. 

6.1.1  LCC  Evaluation  Criteria 

The  following  seven  criteria  assess  the  impact  on  testability  problems  and  were  used  as 
testability  factors  in  the  evaluation  of  the  opportunities: 
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o 

o 

o 

o 

o 

o 

o 


Test  Time  Reduced 

Hard  Fault  Detection 

Hard  Fault  Isolation 

Intermittent  Discrimination  &  Isolation 

False  Alarm  Reduction 

Experience  Level  Reduced 

Reduced  Removals 


Test  Time  Reduced 


Many  -test  programs  must  run  through  the  entire  test  sequence  every  time,  while  providing 
the  technician  with  no  capability  to  make  gross  tests  of  entire  sections,  loop  on  a 
particular  test  sequence,  or  slowly  accumulate  test  information  to  diagnose  intermittents. 
The  use  of  maintenance  expert  systems  is  not  necessary  to  make  the  testing  more  flexible 
and  hierarchical,  but  their  incorporation  produces  this  effect.  A  reduction  in  test  time 
benefits  LCC  by  increased  utilization  of  test  equipment  and  the  test  technicians  for  ATE, 
bench  and  systems  tests.  Large  benefits  in  this  area  could  also  result  in  fewer  ATE 
systems  being  required  at  large  depot  repair  sites. 


Hard  Fault  Detection 

At  every  level  of  testing  it  is  desirable  to  have  the  best  fault  detection  coverage  for  hard 
faults  possible.  This  is  relatively  easy  for  digital  circuits,  but  is  much  more  difficult  for 
circuits  or  systems  which  have  poor  DFT,  use  analog  circuits  or  microprocessors. 
Improving  testability  during  the  design  phase  of  the  electronics  provides  the  greatest 
benefit  for  the  effort  expended,  because  improved  fault  detection  in  the  unit  has  greater 
LCC  benefits  than  improved  fault  detection  solely  based  on  ATE.  As  shown  in  Figure 
3.2-3,  only  by  improving  testability  is  it  possible  to  detect  all  hard  faults. 

Hard  Fault  Isolation 

Because  it  is  a  more  difficult  task  and  a  less  mature  capability,  there  is  more  opportunity 
for  improvement  in  hard  fault  isolation  than  in  their  detection.  Both  can  be  improved 
through  the  use  of  DFT  techniques  and  by  a  maintenance  expert  system  that  can  learn  the 
isolation  strategies  for  real  world  faults.  However,  it  takes  a  greater  level  of  DFT  to 
locate  a  fault  inside  a  complex  network  than  to  detect  a  fault  at  its  output  (see  section 
4.2). 

Intermittent  Discrimination  &  Isolation 

Intermittent  discrimination  and  isolation  is  an  immature  field  which  could  provide 
benefits  to  many  types  of  systems.  The  test  system  must  allow  a  fuzzy  description  of  the 
state  of  a  circuit  (i.e.,  a  description  covering  a  range  of  states:  good,  most  likely 
good— may  have  suspected  as  intermittent,  degrading-but-not-yet-bad,  and  bad).  Products 
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which  accumulate  information  about  an  item's  marginally  bad  performance  during  the 
time  it  is  running  provide  much  more  help  in  detecting  and  isolating  intermittents  than  a 
single  slow  test  of  performance  on  an  ATE  system.  That  approach  also  provides  the  basic 
capability  needed  to  discriminate  between  intermittents  and  false  alarms.  (1) 

False  Alarms  Reduced 

Causes  of  false  alarms  are  most  easily  eliminated  using  CAD  tools  to  help  reduce  false 
alarm  causes  through  better  designs.  CAD  can  provide  the  ability  to  simulate  full  system 
operation  with  BIT  and  software  running  and  to  simulate  various  hard  and  soft  faults.  The 
simulation  results  interpreted  by  the  designer  allow  him  to  find  and  then  eliminate  many 
of  the  causes  of  false  alarms.  Additionally,  methods  to  acquire  and  record  fault 
information  during  circuit  and  system  debug  (in  full  scale  development),  will  find  many 
real  world/environmental  causes  of  false  alarms.  This  information  is  most  easily  acquired 
by  some  form  of  BIT  that  has  non-volatile  memory.  After  it's  fielded,  hardware  becomes 
virtually  unalterable;  however,  the  BIT  test  software  can  still  be  changed  (under 
configuration  control).  Smart  BIT  with  reprogrammable  features  allows  experimentation 
with  proposed  modifications  and  would  also  ease  their  incorporation. 

Experience  Level  Reduced 

Much  of  commercial  and  military  electronic  testing  has  gone  from  bench  to  ATE  so  as  to 
increase  the  reliability  of  testing  and  to  reduce  test  time.  The  military  ATE  software, 
however,  has  greatly  limited  the  technician  in  what  he  can  do.  The  technician  is  rarely 
able  to  modify  the  test  sequence  or  test  limits  to  diagnose  a  problem.  As  circuits  become 
more  complicated,  especially  microprocessors,  there  tends  to  be  little  or  no  information 
available  to  the  test  technician  on  their  operation.  ATE  have  also  become  very  complex 
to  operate  and  each  one  is  different.  It  now  requires  6  to  12  months  of  training  for 
technicians  to  become  familiar  with  operations  of  complex  ATE,  and  learn  how  to 
compensate  for  its  limitations  to  perform  the  testing  and  diagnosis  required. 

Skill  level  reduction  is,  based  on  the  reality  of  military  service  experience,  better 
expressed  as  "experience  level  reduction."  Because  of  this,  it  has  been  the  conclusion  in 
segments  of  the  military  community  (42)  that  ATE  should  be  changed  so  as  to  require  less 
training  for  use  in  performing  maintenance.  As  a  corollary  we  believe  that  the  ATE  must, 
be  configured  to  work  with  the  technician,  rather  than  just  having  him  serve  as  a  button 


pusher.  Instead  of  having  a  computer  guided  probe,  where  the  technician  is  told  where  to 
place  a  probe  so  that  the  computer  can  make  a  test,  there  should  be  interactive  diagnosis 
where  both  the  human  and  the  computer  can  suggest  areas  of  the  circuit  to  be  tested. 
This  is  already  being  done  with  sophisticated  software  (non-AI)  in  some  commercial  card 
testing  ATE,  such  as  that  manufactured  by  Fairchild  (Series  70). 

The  best  product  for  "skill  level  reduction",  (actually  productivity  improvement  using 
short-time  staff),  would  be  a  standard  human  interface  for  ATE  which  would  allow 
interaction  with  the  test  in  progress.  By  providing  an  identical  user  interface  for 
different  ATE,  it  would  also  be  easier  to  reassign  technicians. 

Reduce  Removals 

Removing  an  item  from  a  system  causes  a  whole  chain  of  costly  actions  to  occur.  Many 
authors  address  the  need  to  reduce  removals,  but  the  following  reference  also  includes 
quantitive  evaluations: 

.  .  Field  maintenance  data  (exists)  showing  that  40%  of  the  avionics  equipment 
removed  from  an  aircraft  is  fault  free  and  fully  capable  of  satisfying  its  assigned 
mission  function.  This  inability  to  identify  malfunctioning  equipment  without 
ambiguity  results  in  a  67%  workload  increase  at  organizational  and  intermediate 
maintenance  levels."  -  Top  down  built-in  test  architecture  study,  L.R.  523,  Lock¬ 
heed,  April  1980. 

Using  better  DFT  to  provide  less  isolation  ambiguity  reduces  removals  directly.  Similarly, 
recording  actual  fault  data  can  provide  knowledge  to  technicians  to  allow  their  diagnostic 
experience  to  grow,  provided  that  data  can  be  made  available  to  them  in  a  meaningful  and 
timely  manner. 
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Figure  6.1-1  Distribution  of  Life  Cycle  Cost 

The  following  table  showing  percentage  of  total  life  cycle  cost  for  the  identified  subfactors  was 
derived  from  figure  6.1-1.  These  numbers  will  serve  as  primary  inputs  in  the  evaluation. 

Table  6.1-1  Life  Cycle  Cost  Distribution 

o  Design  3% 

o  Fabrication  21% 

o  Operations  Personnel  7% 

o  Support  Spares  7% 

o  Replacement  Spares  10% 

o  Maintenanace  Personnel  35% 

o  Replacement  Material  5% 


Design 


Designers  are  often  not  aware  that  the  design  phase  is  typically  only  3%  of  the  total  life 
cycle  cost.  Frequently  they  optimize  testing  for  their  own  requirements,  ignoring  field 
types  of  faults  and  minimizing  board  area  allocated  to  testing.  Required  use  of  testability 
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tools  and  standards  will  aid  and  force  the  insertion  of  testability  into  the  design  phase  and 
increase  designer  awareness. 

Fabrication 

AI  applications  can  produce  benefits  in  fabrication  by  reducing  fault  isolation  time  and  by 
improving  fault  detection  coverage.  The  test  time  is  not  as  much  a  concern  as  the 
isolation  time  since  fault  isolation  is  frequently  made  away  from  the  ATE.  Improvement 
in  inherent  testability  of  the  IC,  card,  box,  or  system  all  reduce  fabrication  costs. 
Without  a  high  level  of  inherent  testability,  faults  can  pass  undetected,  and  result  in 
expensive  rework. 

Inherent  testability  improvement  will  also  reduce  costs  associated  with  Automatic  Test 
Program  Generation  (ATPG),  Test  Program  Sets  (TPS),  and  engineering  debugging  at  all 
levels.  During  this  program  phase,  AI  applications  can  also  support  an  aggressive  program 
of  false  alarm  and  intermittent  reduction  by  design  changes  and  modifications  to  the  BIT 
and  system  integrated  test  (STO  that  result  from  capture  and  analysis  of  faults  during  the 
intensive  testing  that  early  production  run  articles  are  subjected  to. 

Operations  Personnel  Costs 

The  largest  benefits  in  Operations  Personnel  Costs  wil  be  achieved  through  improvement 
of  intermittent  isolation  and  by  reduction  in  false  alarm  rate  because  intermittents  and 
false  alarms  greatly  complicate  and  extend  the  tasks  required  of  the  operations  personnel. 
The  benefits  of  hard  fault  ambiguity  reduction  are  of  less  concern.  The  effect  of 
removing  two  boxes  due  to  poor  fault  isolation  capability  is  not  felt  at  this  stage,  but 
rather  at  phases  downstream  in  the  logistics  and  maintenance  system. 

Support  Spares 

Support  spares  are  those  units  purchased  initially  for  the  system  for  support  and 
maintenance  facilities  and  to  fill  the  logistics  pipeline.  Fewer  spares  need  to  be 
purchased  initially  for  a  system  if  it  is  known  that  there  will  be  fewer  removals  caused  by 
false  alarms  and  intermittents  due  to  improved  diagnostic  capability. 


Replacement  Spares 


Replacement  spares  include  cards  and  boxes  purchased  to  replace  the  support  spares 
broken  or  else  destroyed  during  tests,  as  well  as  those  spares  needed  to  replace  units  with 
undiagnosable  problems  of  both  a  hard  and  intermittent  nature.  Improving  fault  isolation 
capability  will  decrease  the  requirement  for  replacement  spares.  Improved  fault  isolation 
also  reduces  the  number  of  units  destroyed  during  test  due  to  "shotgun"  methods  of 
component  replacement  as  an  isolation  technique.  It  is  Boeing's  experience  that  with 
five  or  more  layers  in  a  printed  wiring  board  there  is  a  50%  chance  of  a  layer 
interconnection  being  destroyed  if  the  same  component  is  removed  twice. 

Maintenance  Etersonnel 

The  largest  factor  affecting  a  reduction  in  ATE  maintenance  personnel  cost  is  the 
reduction  in  field  removals.  If  units  are  not  removed,  they  need  not  be  tested,  thus 
reducing  personnel  requirements. 

Repair  Material 

The  factors  effecting  repair  material  are  the  same  as  that  for  replacement  spares,  except 
that  decreasing  intermittents  and  false  alarms  will  not  significantly  decrease  repair 
material.  While  a  false  alarm  can  force  the  replacement  of  a  box,  it  usually  will  not 
cause  a  component  to  be  removed  from  a  board  since  causes  of  false  alarms  are  usually  of 
a  higher  level  than  the  component. 

6.1.3  LCC  Evaluation  Results 


The  AI  applications  of  section  5.0  to  be  evaluated  are: 

o  Computer  Aided  Preliminary  Design  for  Testability— CAPDT  would  be  a 
testability  assistant  directly  available  during  preliminary  design  phases. 

o  Smart  Built-In  Test— Smart  BIT  used  in  boxes  or  cards  can  identify  intermit¬ 
tent  faults  and  reduce  false  alarms. 

o  Smart  System  Integrated  Test— Smart  SIT  is  a  system  level  Smart  BIT  which 
performs  testing  while  the  system  is  operating. 

o  Maintenance  Expert  -  Box— ME  BOX  provides  offline  test  management  with 
self  improvement  of  functional  tests. 


o  Maintenance  Expert  -  System— ME  SYS  describes  the  kind  of  capability  that 
can  be  expected  during  the  next  three  years. 

o  Maintenance  Expert— Smart— ME  Smart  incorporates  the  benefits/risks  in  the  3 
to  5  year  time  period,  of  including  learning  capability  in  the  maintenance 
expert  system  and  its  ability  to  access  to  Smart  BIT  information. 

o  Analog  Automatic  Test  Program  Generation— ATPG-ANA  would  be  able  to 
understand  circuit  functional  operation. 

o  Smart  Bench  is  a  maintenance  expert  system  developed  for  use  with  bench  test 
equipment  controlled  by  an  engineering  workstation. 

o  Digital  Automatic  Test  Program  Generation  with  Machine  Learning— DATPG 
WML  learns  refinements  by  test  engineer  to  DATPG  as  a  result  of  experience. 

Estimates  were  made  of  the  benefit  of  each  AI  application  to  the  testability  factors  in 
Table  6.1-2.  The  maximum  benefit  is  indicated  with  a  10  (example:  Smart  Bit  Benefit  to 
Intermittent  Isolation)  and  the  least  benefit  by  a  zero  (example:  ATPG-analog  effect  on 
false  alarm  reduction). 


Table  6.1-2  Testability  Benefits  of  Opportunity 


TESTABILITY  FACTOR 

CAPD 

SMART 

SMART 

ME 

ME 

ME 

ATP6 

SMART 

DATG 

TEST 

BIT 

SIT 

BOX 

SYS 

SMART 

ANA 

BENCH 

W  ML 

TEST  TIME  REDUCED 

7 

8 

6 

7 

9 

8 

2 

7 

2 

HARD  FAULT  DETECTION 

9 

7 

3 

5 

5 

5 

4 

3 

3 

HARD  FAULT  ISOLATION 

8 

5 

5 

5 

2 

6 

4 

3 

3 

INTERMITTENT  ISOLATION 

9 

18 

9 

4 

2 

9 

8 

5 

2 

FALSE  ALARMS  REDUCED 

4 

9 

9 

2 

2 

9 

8 

2 

2 

EXPERIENCE  LEVEL  REDUCED 

5 

8 

8 

8 

18 

18 

8 

8 

2 

REDUCE  REMOVALS 

8 

8 

18 

3 

6 

8 

3 

3 

3 

Next,  the  benefit  (or  cost)  of  testability  factors  in  each  life  cycle  phase  was  established 
(Table  6.1-3).  The  benefits  range  from  -1  to  +3,  where  -1  indicates  a  net  cost  during  that 
phase,  a  0  indicates  no  LCC  benefit  and  +1,  +2  and  +3  succeedingly  increasing  benefit. 
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Table  6.1~3  Benefit  of  Testability  Factor  in  Each  Life  Cycle  Phase 


TEST 

HARD 

HARD 

INTERN 

FALSE 

EXPER 

REDUCE 

RON 

TIME 

FAULT 

FAULT 

ISOLATE 

ALARMS 

LEVEL 

REMOVAL 

HEIGHT 

REDUCED 

DETECT 

ISOLATE 

REDUCED  REDUCED 

DESIGN 

0 

0 

-1 

-1 

-1 

0 

0 

-3 

FABRICATION 

1 

1 

1 

I 

0 

0 

0 

4 

OPER.  PERSONNEL 

0 

0 

1 

2 

2 

0 

1 

4 

SUPPORT  SPARES 

0 

0 

2 

0 

0 

0 

2 

4 

REPLACE  SPARES 

0 

0 

1 

3 

2 

0 

1 

7 

MAI  NT.  PERSONNEL 

1 

0 

1 

1 

1 

1 

3 

8 

REPAIR  MATERIAL 

0 

0 

i 

0 

0 

0 

1 

2 

The  next  step  of  the  evaluation  consists  of  multiplying  these  two  matrices  (6.1-2,  6.1-3) 


(column-by-row)  to  obtain  the  benefit  of  each  application  to  each  life  cycle  cost  phase. 
The  results  are  shown  in  Table  6.1-4.  For  example,  to  get  "33"  for  a  "Fabrication"  benefit 
for  "CAPD  Test,"  the  "Fab"  row  of  table  6.1-3  is  multiplied  by  the  "CAPD  Test" 
column  in  table  6.1-2  item-by-item.  So,  the  first  four  lines  are  multiplied  by  "1",  the  last 
three  lines  by  "0".  Summing  the  digits:  7+9+8+9  =  33.  This  process  is  repeated  for  each 
matrix  position  to  obtain  table  6.1-4. 


Table  6.1-4  Weighted  Benefit 


CAPD  SMART  SMART 

ME 

ME 

ME 

ATPG  SMART 

DATG 

TEST 

BIT 

SIT 

BOX 

SYS 

SMART 

ANA  BENCH 

H  ML 

DESIGN 

-21 

-24 

-23 

-11 

-6 

-24 

-4 

-12 

-7 

FABRICATION 

33 

30 

23 

21 

18 

28 

10 

20 

10 

OPERATIONS  PERSONNEL 

42 

51 

51 

20 

14 

50 

7 

22 

14 

SUPPORT  SPARES 

32 

26 

30 

16 

16 

28 

14 

16 

12 

REPLACE  SPARES 

SI 

61 

60 

24 

18 

5? 

7 

27 

16 

MAINTENANCE  PERSONNEL 

57 

64 

67 

35 

43 

66 

23 

36 

20 

REPAIR  MAT'L 

14 

13 

15 

8 

8 

14 

7 

8 

4 

The  next  matrix  (6.1-5)  is  a  normalization  of  the  weighted  benefits  shown 
in  Table  6.1-4.  This  eliminates  the  total  weight  bias  (sum  of  column  weights) 
from  each  column  of  Table  6.1-3.  To  do  this,  each  number  of  a  row  is  divided 
by  the  total  weight  of  that  same  LCC  phase  that  was  _ji*ned  at  the  end  of  its 
row  in  Table  6.1-3.  If  the  weighting  had  been  expressed  as  percentages  normali¬ 
zation  step  would  not  have  been  necessary. 

Table  6.1-5  Normalized  Benefit 


To  obtain  the  potential  percentage  of  LCC  reduction  for  each  application,  as  shown  in 
Table  6.1-6,  each  item  in  a  row  of  Table  6.1-5  is  scalar  multiplied  by  its  corresponding 
LCC  distribution  percentage  listed  in  Table  6.1-1.  These  percentages  are  then  summed  to 
get  the  percent  LCC  reduction  per  application. 

Table  6.1-6  Percentage  LCC  Reduction 


CAPO 

SMART 

SMART 

ME 

ME 

ME 

ATP6  SMART 

DATG 

TEST 

BIT 

SIT 

BOX 

SYS  SMART 

ANA  BENCH 

H  ML 

DESIGN 

-.21 

-.24 

-.23 

-.11 

-.06 

-.24 

-.04 

-.12 

-.07 

FABRICATION 

1.73 

1.58 

1.21 

1.10 

in 

• 

1.47 

.53 

1.05 

.53 

OPERATIONS  PERSOM'IEL 

.47 

.60 

.60 

.23 

.1? 

.58 

.08 

.26 

.16 

SUPPORT  SPARES 

.56 

.46 

.53 

.28 

.28 

.4? 

.25 

.28 

.21 

REPLACE  SPARES 

.73 

.87 

.86 

.34 

.26 

.84 

.10 

.3? 

.23 

MAINTENANCE  PERSONNEL 

2.4? 

2.80 

2.?3 

1.53 

1.88 

2.8? 

1.01 

1.58 

.88 

REPAIR  MAT'L 

.40 

.33 

.38 

.20 

.20 

.35 

.18 

.20 

.15 

V.  LCC  REDUCT  I ON/APPL I C 

6.1? 

6.38 

6.26 

3.58 

3.6? 

6.38 

2.8? 

3.63 

2.08 

EFFECT  OF  APPLIC  ON  SYS 

i.ee 

.18 

1.00 

.10 

1.00 

1.00 

.10 

.10 

.10 

X  SYSTEM  LCC  REDUCTION 

6.1? 

.64 

6.26 

.36 

•3.6? 

6.38 

.21 

.36 

.21 

.  vv.v.y- 
V'.y 


CAPO  SMART 

SMART 

ME 

ME 

ME 

ATP6  SMART 

DATO 

TEST 

BIT 

SIT 

BOX 

SYS  SMART 

ANA  BENCH 

M  ML 

K 

-»  • 

DESIGN 

-7.00 

-8.00 

-7.67 

-3.67 

-2.00 

-8.00 

-1.33 

-4.00 

-2.33 

'l 

FABRICATION 

8.25 

7.50 

5.75 

5.25 

4.50 

7.00 

2.50 

5.00 

2.50 

' . 

y-i 

OPERATIONS  PERSONNEL 

7.00 

8.50 

8.50 

3.33 

2.67 

8.33 

1.17 

3.67 

2.33 

j 

SUPPORT  SPARES 

8.00 

6.50 

7.50 

4.08 

4.00 

7.00 

3.50 

4.00 

3.00 

t 

9 

REPLACE  SPARES 

7.2? 

8.71 

8.57 

3.43 

2.57 

8.43 

1.00 

3.86 

2.2? 

»  .  . 

•  .• 

MAINTENANCE  PERSONNEL 

7.13 

8.00 

8.38 

4.38 

5.38 

8.25 

2.88 

4.50 

2.50 

REPAIR  MAT'L 

8.00 

6.50 

7.50 

4.00 

4.80 

7.00 

3.58 

4.00 

3.00 

'  - 

.  .7 

t  • 


*  %*• 

•  /  .  V-W. 
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6.2  TOTAL  EVALUATION  OF  OPPORTUNITIES 


Ten  parameters  are  used  to  assess  the  risk,  cost,  and  benefits  of  each  of  the  AI 
opportunities  to  determine  a  total  evaluation.  Table  6.2-1  shows  each  parameter  with  its 
range  and  weight  Gmportance)  along  with  this  study's  evaluation  of  each  of  the  described 
AI  applications  for  each  parameter.  Table  6.2-2  incorporates  the  parameter  weights  into 
each  evaluation  (by  multiplication  of  every  item  by  its  associated  parameter  weight). 
Those  parameters  which  reduce  the  figure  of  merit  have  weights  which  are  negative.  A 
sum  of  the  weighted  values  for  each  application  is  then  made  to  find  its  figure  of  merit. 

Risk  Demo  3  year:  This  is  the  risk  of  being  able  to  complete  a  demonstration  project  by 
1987.  High  risk  is  generally  due  to  lack  of  research  that  can  be  directly  used  and  is 
shown  by  a  high  numerical  evaluation  (note  the  negative  multiplier).  "Smart"  SIT 
and  maintenance  expert  approaches,  as  can  be  seen  on  evaluation  charts,  are  risky 
for  this  time  frame  but  will  be  less  risky  after  5  years. 

Development  Status;  Applications  which  have  commercially  available  products  are  rated 
as  3.  Prototyped  applications  are  rated  as  a  2,  while  those  with  ongoing 
experimentation/research  are  given  a  one.  Mere  concepts  are  rated  as  a  zero. 

Cost  to  Demonstrate  Product:  This  is  the  estimated  number  of  manyears  necessary  for  a 
contractor  to  make  a  demonstration  of  the  application.  The  evaluation  quantities 
are  based  on  similar  recommendations  in  AI  Applications  to  Maintenance  (3).  Note 
though,  that  the  mere  demonstration  of  an  application  does  not  ensure  it  wide¬ 
spread,  timely  incorporation.  Factors  influencing  this  were  discussed  in  section  5.2. 

Cost  to  Develop  Tool;  Ten  to  thirty  man-years  are  typically  necessary  to  develop  a 
general  purpose  tool  which  can  be  easily  rehosted.  All  applications  are  in  this  range 
except  for  the  Smart  System  level  maintenance  expert,  which  is  more  complex  than 
average. 

Useful  Tool  is  a  Result:  Will  the  AI  technique  be  easily  rehosted  and  used  by  contractors 
and  subcontractors  once  it  is  developed?  The  best  techniques  can  be  directly  used 
by  designers  and  technicians.  Low  ratings  are  given  to  systems  which  require 
specialists  and  much  new  rule  generation. 


Table  6.2-1  Unweighted  Evaluation 


RANGE 

wt 

CAPD 

TEST 

SMART 

BIT 

SMART 

SIT 

ME 

BOX 

ME 

SYS 

ME 

SMART 

ATPO  SMART 

ANA  BENCH 

DATG 

N  ML 

RISK  OEMO  3  YEAR 

8-5 

-I 

2 

3 

3 

8 

8 

3 

3 

3 

3 

DEVELOPMENT  STATUS 

•  +3 

1 

8 

1 

1 

3 

2 

1 

I 

8 

8 

COST  TO  DEMO  PRODUCT 

MYEAR 

-.3 

5 

3 

8 

2 

2 

3 

3 

3 

3 

COST  TO  DEVELOP  TOOL 

MYEAR 

-.2 

31 

It 

28 

28 

28 

48 

38 

38 

18 

USEFUL  TOOL  IS  A  RESULT 

•-3 

1 

3 

4 

8 

3 

3 

3 

3 

3 

3 

RETROFIT  CAPABILITY 

•-3 

.3 

« 

3 

2 

2 

I 

1 

1 

3 

I 

PAYOFF  LEVERAGE  < BEN/COST) 

>20) 

[  .3 

3 

t 

t 

8 

8 

3 

8 

3 

8 

MISSION  CRITIAL  FUNCT 

e-s 

1 

3 

3 

3 

1 

2 

4 

1 

1 

t 

ENABLING  TECHNOLOGY 

e-5 

.3 

5 

3 

3 

8 

8 

8 

8 

8 

8 

LCC  REDUCTION  -  SYSTEM 

1.3 

4.19 

.44 

4.24 

.34 

3.49 

4.38 

.21 

.34 

.21 

Table  6.2-2  Weighted  Evaluation 


SUBTOTAL  OF  WTX  PARAMETRIC 

CAPD 

SMART 

SMART 

ME 

ME 

ME 

ATPG 

SMART 

DATG 

TEST 

BIT 

SIT 

BOX 

SYS 

SMART 

ANA 

BENCH 

N  ML 

RISK  DEMO  3  YEAR 

-2 

-3 

-3 

8 

8 

-3 

-3 

-3 

-3 

DEVELOPMENT  STATUS 

8 

1 

1 

3 

2 

1 

1 

8 

8 

COST  TO  DEMO  PRODUCT 

-1.3 

-.9 

-2.4 

-.4 

-.4 

-1.5 

-1.3 

-.9 

-.9 

COST  TO  DEVELOP  TOOL 

-4 

-2 

-4 

-4 

-4 

-8 

-6 

-4 

-2 

USEFUL  TOOL  IS  A  RESULT 

3 

4 

8 

3 

3 

3 

3 

3 

3 

RETROFIT  CAPABILITY 

8 

.9 

.4 

.4 

.3 

.3 

.3 

1.3 

.3 

PAYOFF  LEVERAGE  (BEN/COST) 

2.3 

8 

8 

8 

8 

2.3 

8 

2.3 

8 

MISSION  CRITIAL  FUNCT 

3 

3 

3 

1 

2 

4 

1 

1 

1 

ENABLING  TEONOLOCY 

2.3 

2.3 

1.3 

8 

8 

8 

8 

8 

8 

LCC  REDUCTION  -  SYSTEM 

9.3 

1 

9.4 

.3 

3.3 

9.4 

.3 

.3 

.3 

FOM  -  SUM  WEIGHTED  PARAMETERS 

12.8 

4.3 

4.1 

3.3 

8.2 

3.9 

-4.9 

.4 

-3.3 

114 


Retrofit  Capability:  High  ratings  are  given  to  techniques  which  can  be  cost  effectively 
applied  to  already  fielded  systems.  Smart  Bench  can  be  applied  with  no  modifica¬ 
tion  to  fielded  units.  Smart  BIT  can  be  used  in  the  retrofit  of  existing  BIT  systems 
since  its  chip  set  would  require  little  room  and  its  ability  to  record  faults  would  not 
require  any  new  system  or  logistics  capability. 

Payoff  Leverage:  Any  application  that  has  a  potentially  high  payoff  (benefits  more  than 
20x  the  development  cost)  is  given  additional  weight.  As  a  first  approximation,  a 
weight  of  5  was  chosen  to  indicate  that  the  20x  threshold  is  met  or  exceeded  by  an 
application. 


Mission  Critical  Function:  Is  the  system  able  to  perform  its  mission  when  called  upon? 
If  the  system  is  unavailable  due  to  maintenance,  readiness  is  decreased.  The  worst 
case  situation  comes  during  a  war  time  situation.  Maintenance  actions  of  4  hours 
which  are  normally  accepted  during  peacetime  operation  may  be  considerably 
lengthened  in  a  high  sortie  generation  schedule.  An  application  is  able  to  help  a 
system  perform  its  mission  well  (a  rating  of  5)  if  it  provides  a  high  level  of  self  test 
and  has  a  future  capability  to  perform  prognostics,  all  while  assuring  a  low  rate  of 
Retest  OK  for  removed  boxes.  Figure  6.2-1  shows  some  of  the  factors  that 
determine  readiness  from  Meth's  article  in  the  Proceedings  of  the  Reliability  & 
Maintainability  Symp.  1984.  (47) 


Typical 

Readiness 

Measure 


Operational 
Availability  (A0)  = 


MTBF 


MTBF  +  MTTR  +  MLDT 


MTBF  =  Mean  Time  Between  Failure 
MTTR  =  Mean  Time  to  Repair 
MLDT  s  Mean  Logistics  Downtime 


Figure  6.2-1  Factors  That  Effect  Weapon  System  Readiness  (47) 
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Enabling  Technology:  Those  applications  that  are  going  to  be  essential  to  broad 
reductions  in  testability  problems  in  the  near  term  Are  given  a  rating  of  5,  important 
applications  are  given  a  3. 

LCC  Reduction:  The  data  from  Table  6.1-6  is  used  here. 

FIGURE  OF  MERIT 

The  bottom  line  of  table  6.2-2  is  the  figure  of  merit  for  each  application  evaluated.  It  is 
a  balance  between  the  large  negative  factors  of  risk  of  demonstration  in  3  years  and  cost 
to  develop  a  tool,  against  the  substantial  positive  weights  given  to  having  a  tool  and  the 
amount  of  life  cycle  cost  reduction.  Figures  of  merit  with  a  range  from  12.8  to  -6.9  were 
separated  into  three  categories,  with  at  least  a  4  point  gap  between  the  high,  medium,  and 
low  recommendations. 


Category 

FOM 

Application 

1 

12.8 

Computer  Aided  Preliminary  Design  For  Testability 

2 

8.2 

Maintenance  Expert  -  System 

6.5 

Smart  BIT 

6.1 

Smart  System  Integrated  Test 

5.9 

Maintenance  Expert  -  Smart 

3.5 

Maintenance  Expert  -  Box 

3 

.6 

Smart  Bench 

-3.3 

Digital  Automatic  Program  Gen.  with  Machine  Learning 

-6.9 

Automatic  Test  Program  Generation  -  Analog 

Category  1  represents  the  highest  level  of  recommendation;  it's  practical,  it  has  large 
benefits.  Category  2  represents  some  excellent  candidate  applications,  but  each  has  some 
shortcoming  that  either  makes  it  more  risky  or  is  of  a  lesser  benefit,  based  on  cost. 
Category  3  items  provide  insufficient  benefit  for  their  cost  to  be  considered.  These 
ratings  and  other  more  intangable  factors,  that  influence  the  acceptability  of  an  approach 
are  discussed  further  in  Section  7.0,  Study  Findings  and  Recommendations. 


7.0  STUDY  FINDINGS  AND  RECOMMENDATIONS 

This  study  provides  the  foundation  for  a  logical  and  cost-effective  program  to  apply 
artificial  intelligence  to  testability.  Emphasis  was  on  those  artificial  intelligence  (AI) 
techniques  capable  of'  practical  application  to  testability  problems  immediately  or  with 
low-risk  development  within  three  to  five  years. 

It  was  shown  in  this  study  that  cost  effective  testability  techniques  for  use  in  electronic 
systems  can  utilize  artificial  intelligence  techniques.  Testability  problems  addressed 
include  reducing  false  alarms,  isolating  intermittent  failures,  assuring  adequate  failure 
coverage  by  Built-In  Test  and  Automatic  Test  Equipment  (BIT/ATE)  and  reducing 
maintenance  training  requirements.  In  addition,  these  techniques  should  be  capable  of 
reducing  life  cycle  costs  by  approximately  5%  by  reducing  testing  costs,  which  account 
for  more  than  20%  of  LCC. 

It  was  determined  that  the  most  cost  effective  approach  to  development  and  use  of 
application  tools  to  develop  diagnostic  rules  which  are  system  independent  (metarules). 
It  was  recommended  that  this  approach  be  used  for  any  development  program  applying  an 
expert  system  to  testability.  It  was  also  recommended  that  new  development  in  applying 
AI  to  testability  use  standard  engineering  work  stations  as  the  engineer’s  development 
tool  because  the  aerospace  industry  is  now  racing  toward  massive  use  of  engineering 
work  stations  in  its  design  organizations.  Also,  AI  languages  such  as  LISP  and  PROLOG 
should  also  be  available  shortly  on  the  work  stations. 

This  section  consolidates  and  presents  a  detailed  discussion  of  the  fundings  and 
recommendations  of  this  study. 

7.1  STUDY  FINDINGS 

The  study  findings  described  in  this  section  are  arranged  in  the  same  order  as  this  study 
report  for  ease  of  reference.  These  include  findings  in  the  following  areas: 

o  AI  techniques 

o  ATE/BIT  applications 

o  Testability  applications 

o  AI  solutions  to  testability  problems 

o  General  findings 
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7.1.1  AI  TECHNIQUE  FINDINGS 


Expert  system  development  is  the  most  active  area  in  artificial  intelligence.  These 
systems  are  now  capable  of  being  used  as  knowledge  managers  with  several  testability 
applications. 

Diagnostic  metarules  and  machine  learning  will  allow  maintenance  expert  system  tools 
to  be  developed  that  can  be  easily  applied  to  many  projects.  Presently,  most  expert 
systems  are  developed  through  the  knowledge  engineering  lifecycle  for  each  project  with 
the  knowledge  coming  from  a  single  human  expert.  A  few  maintenance  expert  system 
tools  have  been  created  (8,  9),  including  some  that  can  incorporate  diagnostic  experience 
(10,11).  The  use  of  system  independent,  diagnostic  metarules  will  permit  a  maintenance 
expert  system  to  be  created  much  more  economically  than  those  previously  developed. 
This  will  reduce  the  time  to  create  an  expert  system  for  maintenance  from  5  manyears 
to  about  1  manyear  for  commercial  systems.  A  similar  five  times  reduction  in  time  for 
military  systems  might  also  be  possible,  but  the  added  overhead  and  documentation 
required  for  military  programs  may  reduce  this  ratio. 

7.1.2  BIT /ATE  FINDINGS 

The  two  primary  methods  of  testing  military  electronics,  Built-In  Test  (BIT)  and 
Automatic  Test  Equipment  (ATE),  have  reached  sophisticated  levels  of  capability  but 
still  require  excessive  manpower  for  maintenance.  A  survey  was  made  of  current 
applications  of  AI  in  BIT  and  ATE  as  well  as  BIT  and  ATE  problems  in  order  to  determine 
the  potential  for  near  term  application  of  artificial  intelligence.  Information  was 
obtained  from  published  literature  and  reports,  interviews  with  ATE  vendors  and  relevant 
R&D  groups  in  government  and  industry. 

The  key  BIT /ATE  findings  show  a  lack  of  uniform  application  of  design  for  testability 
(DFT)  techniques: 

o  Present  application  of  AI  to  BIT  and  ATE  is  practically  nonexistent 
o  Designers  are  not  given  incentive  or  feedback  on  DFT  for  BIT /ATE 
o  There  is  inadequate  management  visibility  of  DFT  quality 
o  ATE  problems  are  generally  due  to  lack  of  adequate  DFT  and  BIT 


o  ATE  test  problems  are  increased  because  BIT  data  is  often  ignored 
o  BIT  experts  are  needed  since  BIT  is  frequently  designed  new  for  each  military 
acquisition. 

These  findings  resulted  in  the  recommendation  of  applying  A1  to  increase  inherent 
testability  by  the  use  of  an  expert  consulting  system  (Computer  Aided  Preliminary 
Design  for  Testability)  that  can  be  used  during  preliminary  design  and  by  use  of  a  Smart 
BIT  chip  set. 

7.1.3  TESTABILITY  APPLICATIONS 

Five  testability  improvement  areas  were  found  to  have  a  potential  for  near  term 
application  with,  or  as  part  of  an  AI  approach. 

1)  Self  improving  diagnostics: 

o  Functional  test  sequences  can  be  cost  effectively  improved, 
o  Incorporation  of  diagnostic  experience  through  automated  learning  is  a 
promising  near-term  concept. 

2)  More  effective  fault  detection  and  isolation: 

o  Better  Design  For  Testability  is  the  most  cost  effective  means, 
o  Smart  BIT  provides  the  only  means  to  capture  fault  data  in  the  field. 

3)  Discrimination  between  false  alarms  and  intermittent  faults: 

o  This  capability  will  take  at  least  five  years  to  evolve.  Information 
necessary  is  not  yet  captured  (by  Smart  BIT)  and  AI  analysis  techniques 
are  not  yet  developed. 

4)  Reduction  of  skills  required  for  maintenance: 

o  AI  applied  to  information  processing  can  reduce  specific  experience 
needed  by  a  maintenance  technician. 

5)  Integrated  Diagnostics: 

o  AI  can  provide  cost  effective  information  processing  for  the  system, 
ATE  and  bench  testing. 


7.1.*  AI  SOLUTIONS  TO  TESTABILITY  PROBLEMS 
Technology  Breakthroughs 

Two  major  breakthroughs  have  occurred  that  will  have  wide  reaching  effects  on  the 
productivity  of  maintenance  expert  system  applications  and  on  both  developer  and  user 
interfaces.  These  are  the  development  of  diagnostic  metarules  for  expert  systems  and 
the  proliferation  of  engineering  work  stations. 

Typical  useful  expert  systems  presently  require  more  than  5  manyears  to  develop.  In 
order  to  create  maintenance  expert  systems  quickly,  they  must  be  built  from  generic 
components  so  that  a  large  portion  of  the  software  can  be  reused  on  each  implementa¬ 
tion.  Corporate  developments  at  GE  and  DEC  have  created  expert  systems  using  these 
techniques  which  intelligently  manage  test  sequences  and  can  be  adapted  to  a  new 
system. 

Dr.  Pau's  work  at  Battelle  has  developed  400  application  independent  diagnostic  meta¬ 
rules  which  have  already  been  applied  to  several  experimental  systems.  These  metarules 
include  a  circuit  network  that  can  describe  many  types  of  real  electronic  faults.  This 
provides  a  breakthrough  both  in  reducing  the  costs  and  increasing  the  capability  of 
maintenance  expert  systems. 

The  engineering  work  station  provides  a  common  host  for  development  of  all  of  the 
recommended  Al  applications.  Used  in  conjunction  with  the  rule  structure  of  expert 
systems,  it  permits  a  synergistic  development  of  the  requirements  common  to  many  of 
the  applications  (diagnostic  rules,  network  understanding,  graphical  display,  etc.). 

Engineering  work  stations  are  gaining  widespread  acceptance  in  the  aerospace  industry. 
There  are  multiple  sources  for  both  hardware  and  software  that  make  work  stations  big 
enough,  fast  enough,  and  cheap  enough  to  be  really  useful.  They  will  provide  direct, 
personal  access  to  AI  applications  by  the  electronics  designers.  As  a  result,  the  majority 
of  the  end  users  will  be  able  to  make  use  of  applications  in  engineering  work  stations. 

AI  Applications  Evaluated 

Nine  potential  AI  applications  were  developed  from  the  study  survey  findings  and. 
evaluated  with  respect  to  testability  problem  areas.  Table  7.1-1  is  an  applicability 
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matrix  that  shows  this  comparison  in  chart  form.  Details  supporting  these  findings 
follows  in  summary  form: 

Computer  Aided  Preliminary  Design  for  Testability  (CAPDT)  would  be  a  testability 
assistant  directly  available  during  preliminary  design  phases. 

Smart  Built-In  Test  (Smart  BIT)  used  in  boxes  or  cards  can  identify  intermittent  faults 
and  reduce  false  alarms. 

Smart  System  Integrated  Test  (Smart  SIT)  is  a  system  level  Smart  BIT  which  performs 
testing  while  the  system  is  operating. 

Maintenance  Expert  -  Box  (ME  BOX)  provides  offline  test  management  with  self 
improvement  of  functional  tests. 

Maintenance  Expert  -  System  (ME  SYS)  describes  the  kind  of  capability  that  can  be 
expected  during  the  next  three  years. 

Maintenance  Expert— Smart  (ME  Smart)  incorporates  the  benefits/risks  in  the  3  to  5  year 
time  period,  of  including  learning  capability  in  the  maintenance  expert  system  and  its 
ability  to  access  to  Smart  BIT  information. 

Automatic  Test  Program  Generation  (ATPG)  would  be  able  to  understand  circuit  func¬ 
tional  operation;  however  this  application  has  the  lowest  figure  of  merit. 

Smart  Bench  is  a  maintenance  expert  system  developed  for  use  with  bench  test 
equipment  controlled  by  an  engineering  workstation. 

Computer  Aided  Preliminary  Design  for  Testability  (CAPDT) 

The  findings  related  to  this  application  are: 

o  Testability  experts  are  needed  during  initial  design  to  meet  new  requirements 

o  Not  enough  human  testability  experts  are  available 

o  An  AI  testability  expert  can  be  made 


Table  7.1-1  AI  Applications  to  Testability  Problems 


improvements.  Smart  BIT  can  improve  diagnostic  techniques  of  the  programmable- 
recording  BIT  using  a  com  pa  table  chip  set.  Signal  analysis  similar  to  a  technician's 
approach  can  also  be  used  by  Smart  BIT. 

7.1.5  AI  CAN  IMPROVE  INFORMATION  CAPTURE,  FLOW  AND  PROCESSING 

The  IDA  report  Vol.  Ill  (45)  recognized  that  poor  information  flow  prevents  improve¬ 
ments  of  testing  in  the  field. 

"Diagnostics  long  term  trend  analysis  could  not  be  supported  by  information  from 

field  data  systems . Accuracy  and  availability  of  field  data  in  most  cases  was 

judged  to  be  insufficient  to  support  requirements,  development,  short  term  trend 
analysis,  and  timely  problem  identification." 

A  majority  of  the  AI  applications  focus  on  processing  the  information  associated  with 
testing.  The  performance  of  the  application  is  a  function  of  the  amount  of  information 
available  to  it.  Increasing  testability  improves  the  AI  application  performance,  cost- 
effectiveness,  ease  of  creation,  etc. 

Figure  7.1-1  presents  current  testability  information  flow  paths.  Current  barriers  to 
information  flow  which  have  the  potential  of  being  reduced  by  AI  applications  are: 

o  BIT  too  expensive  to  redesign  to  reduce  false  alarms 

o  BIT  (and  flight  records)  do  not  capture  environmental  information  associated 
with  alarms  to  permit  later  false  alarm  analysis 
o  Designer  is  not  aware  how  well  he  is  providing  for  DFT 

o  Operator/maintenance  personnel  not  given  feedback  on  their  diagnostic 
decisions,  nor  is  their  diagnostic  experience  preserved 
o  Fixed  test  sequences  on  ATE  have  resulted  in  long,  inflexible  testing  because 
test  sequence  information  usually  not  provided  by  the  test  requirements 
document 
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Use  of  some  of  the  AI  applications  described  in  this  document  have  the  potential  to 
significantly  improve  the  information  flow.  This  is  shown  in  figure  7.1-2. 
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Figure  7.1-1  Current  Information  Flo w  Is  inadequate  to  Improve  Testability 


Figure  7. 1-2  Al  and  Engineering  Work  Station  Improves  Information  Flow 


7.2  STUDY  RECOMMENDATIONS 


The  developments' in  expert  systems  and  especially  diagnostic  metarules  permits  the 
development  of  a  maintenance  expert  system  template  which  could  be  quickly  applied  to 
maintenance  of  many  different  systems.  These  same  diagnostic  metarules  could  be  used 
for  system  and  box  level  maintenance  experts  working  in  real  time,  on  ATE  or  on  the 
bench.  Diagnostic  rules  developed  by  different  organizations  could  be  shared  if  they 
used  standard  systems.  The  limited  focus  of  testability  expert  systems  permits  the 
creation  of  reusable  software. 

The  engineering  work  station  provides  a  host  which  can  be  used  for  the  development  of 
all  of  the  applications  shown  in  section  7.1  and,  in  addition,  can  be  the  host  for  use  of 
many  of  the  applications.  These  work  stations  are  becoming  a  standardized  industry 
approach  and  a  means  to  get  the  resulting  A!  application  tool  to  the  electronics  designer 
as  well  as  a  standard  for  sharing  AI  techniques  and  data  between  different  testability 
research  and  development  organizations. 

Operator  interface  standards  (test  and  graphic)  can  be  developed  to  reduce  retraining 
and  relearning  by  developers,  engineers,  and  technicians.  Data  exchange  standards  are 
essential  for  developing  a  national  testability  library  of  design  examples  and  for  reusable 
functional  description  of  some  integrated  circuits. 

Without  a  cooperative  development  of  the  AI  opportunities,  it  will  be  many  years  until 
each  aerospace  contractor  develops  cost  effective  tools  to  incorporate  testability  into 
the  designs.  Human  testability  experts  do  not  exist  in  sufficient  quantities  to 
incorporate  the  desires  of  the  new  military  testability  standards.  The  AI  testability 
assistant  evaluated  in  this  study  appears  to  be  the  only  method  available  to  incorporate 
all  of  the  necessary  test  considerations  into  the  early  project  phases.  The  military 
testability  standard  does  not  state  how  testability  is  to  be  achieved,  instead,  it  requires 
audits  of  testability  after  each  phase  of  development. 

The  recommended  evolution  of  development  of  AI  opportunity  is  shown  in  figure  7.2-1. 
The  relative  cost  of  each  program  is  proportional  to  the  area,  with  the  metarules  being 
approximately  a  one  million  dollar  development  effort.  This  shows  a  possible  composite 
program  chart  which  assumes  a  synergistic  environment,  coordinated  between  developers 
and  end  users.  Its  major  features  are; 


Figure  7.2-1  Recommended  Evolution  of  AI  Opportunities  In  Testability 


o  Develop  diagnostic  metarules  on  engineering  work  stations 
o  Enhance  metarules  with  self  improving  diagnostic  capability 
o  Provide  resulting  maintenance  expert  as  a  tool 

o  Develop  recording-reprogrammable  BIT  chip  set  to  gather  critical  fault 
information 

o  Later,  enhance  the  chip  set  with  Smart  BIT  capabilities,  and  provide  it  as  tool 
which  can  be  used  by  contractors 

o  Develop  Computer  Aided  Preliminary  Design  For  Testability  which  is  crucial 
to  early  testability  insertion 

o  Develop  a  national  library  of  testability  examples  and  rules  which  can  be  used 
on  the  engineering  work  station 

By  coordinating  the  use  of  the  same  components  in  the  development,  substantial  time 
and  cost  is  saved: 

o  Same  data  interchange  format  for  rule  and  data  sharing 
o  Same  AI  language 

o  Same  Rule  format  for  Diagnostics— Metarules  are  System  Independent 

.  Has  potential  for  low  cost  AI  application  development  for  each  system 
o  Shared  AI  technique  development 

o  Same  operator  interface  for  all  applications 

.  Reduces  learning  time  for  engineers  and  technicians 
.  Integrated  information  display 
o  Same  System  and  Circuit  description  as  that  used  by  designers 
on  their  engineering  work  stations 

AI  opportunities  permit  the  acceleration  of  testability  insertion  by  the  aerospace 
contractors  by  providing  them  with  useful  tools  years  before  they  would  otherwise  be 
developed.  The  current  amount  of  LCC  devoted  to  testing  is  25%.  When  the  new 
testability  standards  are  imposed  this  might  be  expected  to  increase  a  few  percentage 
points  into  the  early  90's  while  the  contractors  each  develop  Smart  BIT,  testability 
assistants,  maintenance  experts,  etc.  Then  it  is  predicted  that  the- costs  associated  with 
testability  would  drop  by  at  least  5%,  to  less  than  25%. 

The  cost  of  this  study's  recommended  development  program  (figure  7.2-1)  is  approxi¬ 
mately  $10M.  The  cost  of  avionics  each  year  is  $10B.  The  acceleration  of  testability 
insertion  could  result  in  an  avionics  cost  savings  of  at  least  2%  per  year  ($200M)  over  the 


independent  development  of  testability  tools.  Over  a  5  year  period  this  amounts  to  a  $1B 
cost  savings  for  $10M  investment. 

Recommendations  to  Industry 

There  are  many  paths  to  incorporating  AI  in  a  product.  The  longest  path  is  to  ask  an 
engineer  to  use  the  AI  language  on  an  existing  general  purpose  computer  to  make  an 
expert  system.  This  approach  will  take  many,  many  man  years  to  get  beyond  the  trivial 
(50  rule)  stage.  There  are  an  assortment  of  commercially  available  systems  for 
producing  expert  systems  which  eliminate  reinvention  of  the  wheel. 

While  it  is  currently  costly  to  develop  the  rules  of  an  expert  system  by  extracting  them 
from  experts,  soon  developments  similar  to  the  Boeing  Computer  Services  Expertise 
Transfer  System  will  reduce  the  need  for  knowledge  engineers  in  creating  an  expert 
system . 
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For  development  of  maintenance  expert  systems,  participation  with  Dr.  Pau  is 
recommended.  The  participation  cost  for  each  sponsor  is  in  the  range  of  $10,000  to 
$20,000  for  nine  months.  (Battelle,  Geneva  Research  Centres;  7,  route  de  Grieze;  1227 
Carouge;  Geneva,  Switzerland  or  Dr.  Barry  Brunstein.,  Battelle  -  Digital  Systems  and 
Technology  Section,  505  King  Ave,  Columbus,  Ohio,  43201). 
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Corp.,  Final  Report,  August  31,  1981.  Available  from  Bill  Rizzo,  Navy  Training 
Equipment  Center.  No  DoD  publication  number. 

26.  Diagnosis  Using  Hierarchical  Design  Models,  Michael  R.  Genesereth,  178-283, 
AAAI-82.  (8.2,  pg  156) 

27.  Representing  Structure  and  Behavior  of  Digital  Hardware,  Randall  Davis,  Howard 
Shrobe,  75-82,  Computer,  October  1983.  (8.2,  pg  141) 

28.  Computers  That  Think  Like  Experts,  Paul  Kinnucan,  High  Technology,  January 
1984,  pg.  30-42. 

29.  Adaptive  Fault  Isolation  With  Learning,  W.  R.  Simpson,  J.  R.  Agre,  IEEE 
AUTOTESTCON,  1983,  pg.  331-335.  (8.2,  pg  141) 

30.  Logistical  Impacts  Within  the  Cost  Analysis  Community,  Robert  E.  Lee,  Spring 
1983,  Armed  Forces  Comptroller.  (8.2,  pg  164) 

31.  Conference  on  Integrated  Diagnostics,  National  Security  Industry  Association: 
Automatic  Test  Committee  and  Logistics  Management  Committee,  February  1983. 
(8.2,  pg  157) 


32.  Design  for  Testability  Using  Logic  Programming,  Paul  W.  Horst  man,  19S3  Inter. 
Test  Conf.,  pg.  706-713.  (8.2,  pg  162) 

33.  HITEST  -  Intelligent  Test  Generation,  Gordon  D.  Robinson,  IEEE  International  Test 
Conf.  1983,  pg.  311-323. 

34.  Design  For  Testability:  An  Integrated  Approach  to  VLSI  Testing,  Tulin  Erdim 
Mangir,  IEEE  International  Conference  on  Computer-Aided  Design,  September 
1983.  (8.2,  pg  161) 

35.  A  Merger  of  CAD  and  CAT  is  Breaking  The  VLSI  Test  Bottleneck,  Hnatek 
Electronics,  April  19,  1984. 

36.  Building  Expert  Systems,  F.  Hayes-Roth,  D.A.  Waterman,  D.B.  Lenat,  Addison- 
Wesley  Teknowledge  Series  Vol  1, 400  pages.  (8.2,  pg  138) 

37.  The  Application  of  Artificial  Intelligence  To  A  Maintenance  and  Diagnostic 
Information  System  (MDIS),  1983,  D.  R.  Antonelli,  Proceedings  of  the  Joint 
Services  Workshop  On  Artificial  Intelligence  and  Maintenance,  Boulder,  Co.,  To  be 
published.  (8.2,  pg  142) 

38.  Automation  In  Design  for  Testability,  Agrawal,  Custom  IC  Conf.,  May  1984.  (8.2, 
Pg  155) 

39.  Reliability,  Availability,  and  Maintainability  (RAM),  October  1982,  Navy  Research 
Advisory  Committee.  82-2.  (8.2,  pg  140) 

40.  Professional  Report:  Computer-Assisted  Education:  What's  Not  Happening?, 
Francis  Fisher,  Journal  of  Computer-Based  Instruction,  Summer  1982.  (8.2,  pg  147) 

41.  Intelligent  Tutoring  System,  Edited  by  D.  Sleenan  and  J.  S.  Brown,  1982.  (8.2, 

pg  147) 

42.  ATE— Applications  of  Artificial  Intelligence,  A.  J.  Kunert,  1982,  IEEE 
aUTOTESTCON.  (8.2,  pg  153) 
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43.  Applying  Knowledge  Engineering  to  TPS  Development,  Thomas  G.  Freund,  IEEE 
AUTOTESTCON,  1983,  pg.  318-321.  (8.2,  pg  154) 

44.  Maintainability  Prediction  Revisited:  Diagnostic  Behavior,  System  Complexity, 
and  Repair  Time,  Joseph  Wohl,  IEEE  Transactions  on  Systems,  Man  and  Cyber¬ 
netics,  May/June  1982,  pg.  241-250.  (8.2,  pg  140) 

45.  R&M  Study  of  IDA  Vol  ID,  August  1983,  Institute  of  Defense  Analysis,  HIGH  PAY¬ 
OFF  AREAS  part  7.0.  (8.2,  pg  141) 

46.  The  Need:  Improved  Diagnostics— Rather  than  Improved  Reliability,  Proceedings 
Annual  Reliability  and  Maint inability  Symposium  1984,  3.  G.  Malcolm,  G.  L. 
Foreman,  Jan  1984,  pg  315.  (8.2,  pg  153) 

47.  The  Operational  Readiness:  The  Reliability  and  Maintainability  Connection,  Martin 
Meth,  Kurt  Greene,  1984  Proceedings  Reliability  and  Maintainability  Symposium, 
pg.  83-85. 
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8J  ANNOTATED  REFERENCE  "BIBLIOGRAPHY" 

This  annotated  bibliography  of  Artificial  Intellengence  afnd  Testability  subjects  includes 
many  references  used  in  the  AI  Applications  to  Testability  report,  those  which  are 
referred  to  in  the  report  by  numbered  reference  are  in  section  8.1. 

Page  _ Subject _ 

156  THEOREM  PROVING 

1 37  NATURAL  LANGUAGE,  VOICE,  SPEECH 

138  DATA  BASE  RETRIEVAL/REPRESENTATION 

138  EXPERT  SYSTEMS 

140  MAINTENANCE  EXPERT 

143  ROBOTICS 

144  AUTOMATIC  PROGRAMMING 

1 44  VISION 

144  PATTERN  RECOGNITION 

145  CAD 

147  MAINTENANCE  TRAINING 

147  MACHINE  LEARNING 

149  AI  BOOKS 

150  MAGAZINES  and  CONFERENCES 

151  AI  MISC. 

153  BIT/ ATE 

155  DIAGNOSIS/FAULT  DETECTION 

159  TESTABILITY 

163  INTERMITTENT  FAILURES 

164  COSTS,  LCC 

—TP - THEOREM  PROVING - 


Witold  S.  Wojciechowski  and  Anthony  S.  Wojcik,  Automated  Design  of  Multiple-Valued 
Logic  Circuits  by  Automatic  Theorem  Proving  Techniques,  IEEE  Transactions  on 
Computers,  September  1983 

Wolfgang  Bibel,  A  Comparative  Study  of  Several  Proof  Procedures,  Artificial 
Intelligence,  1982,  pg  269-293 

Three  algorithms  for  testing  the  complementarity  of  a  matrix  (representing  a  propo¬ 
sitional  formula)  are  developed  in  stages.  Any  of  these  algorithms  is  distinguished  from 
its  predecessor  by  a  specific  feature  (linearity,  jump,  non-normal  form)  which  endows  it 
with  a  provable  advantage  with  respect  to  its  performance. 

Neil  V.  Murray,  Completely  Non-Causual  Theorem  Proving,  Artificial  Intelligence,  3 an 
1982,  pg  67-85. 

The  proof  procedure  operates  on  quantifier-free  formulas  of  the  predicate  calculus  which 
are  not  truth-functionally  normalized  in  any  way. 

David  E.  Wilkins,  Using  Knowledge  to  Control  Tree  Searching,  Artificial  Intelligence,  Jan 
1982,  pg  1-51. 

How  to  limit  the  search  without  a  depth  limit  or  any  other  artificial  effort  limit. 

Dennis  De  Champeaux,  Bidirectional  Heuristic  Search  Again,  Journal  of  the  ACM,  30, 
Jan  1983,  pg  22-32. 


Extensive  modifications  of  a  previously  published  bidirectional  heurisitc  search  algorithm 
are  presented  in  order  to  preserve  the  property  that  shortest  solutions  are  found  under 
appropriate  circumstances. 


Vipin  Kumar,  Laveen  N.  Kanal,  Branch  and  Bound  Formulations  for  And/Or  Tree  Search 
with  Applications  in  Pattern  Recognition  and  Game  Playing,  Proceeding  of  6th  Interna¬ 
tional  Conference  on  Pattern  Recognition.  19-22  Oct  1982,  pg  699-704. 

We  briefly  describe  an  abstract  model  of  Branch  and  Bound  which  incorporates  the 
essentials  of  many  heuristic  search  procedures  developed  in  AI  for  searching  state-space 
graphs  AND/OR  graphs. 


— NL - NATURAL  LANGUAGE  VOICE  SPEECH 


William  Gevarter,  An  Overview  of  Computer  Based  Natural  Language  Processing,  72, 
April  1983,  NBS,  NBSIR  83-2687 

Excellent  report  is  part  of  NBS/NASA  overview  reports. 


Terry  Winograd,  Language  as  a  Cognitive  Process  Voll:  Syntax,  Addison-Wesley,  1983, 
621+,  Stanford  University 

Chapters:  Viewing  Language  as  a  KB  Process  Word  Patterns  +  Classes 
Contex  Free  Grammers  and  Parsing  Transformational  Grammar 
Augmented  Transition  Network  Grammars  Feature  and  Function  Grammars 
Computer  Systems  for  NL  Parsing 


3ean-Paul  Haton,  Speech  Recognition  and  Understanding,  1982  IEEE  Pattern  Recogni¬ 
tion.  570-581 

Technical  overview  paper  with  details 


Phil  Hayes,  Eugene  Ball,  Raj  Reddy,  Breaking  the  Man-Machine  Communication  Barrier, 
March  1981,  IEEE  Computer,  19-30, 

Revised  and  extended  version  of  "Computers  with  Natural  Communication  Skills",  in 
Computer  Science  Research  Review  1980-81,  Carnegie-Mellon  University 
To  breach  the  man-machine  communication  barrier,  our  program  of  research  combines  a 
rather  general  analytic  approach  with  a  practical  application  of  those  general  analyses. 
The  analytic  aspect  of  our  work  involves  observation  of  human  communications  with  both 
people  and  computer  systems. 


Magazines  devoted  to  voice:  Voice  News,  Voice  Processing,  Speech  Technology. 


Alistair  D.C.  Holden,  Paul  V.  Haley,  Adaptive  Methods  in  Speech  Understanding  Systems, 
IEEE  1982  Cybernetics  and  Society,  28-30  October  1982,  161-165 


Adaptation  is  descussed  at  various  levels,  namely,  (a)  at  the  level  where  an  isolated  word 
recognizer  (IWR)  can  adapt  to  gradual  changes  in  the  speaker's  voice,  (b)  where  an  IWR 
can  detect  large  changes  in  the  voice  due  to  stressful  situations  and  adapt  to  the 
changes,  and  (c)  where  the  system  as  a  whole  can  improve  with  experience. 


W.  G.  Lehnert,  M.  H.  Ringle,  Strategies  for  Natural  Language  Processing,  1982,  533, 
Erlbaum, 

Some  Authors  and  Chapters  are  as  follows 

Riesbeck  Realistic  Language  Comprehension 

DeJong  An  Overview  of  the  FRUMP  System 

Hobbs  Towards  an  Understanding  of  Coherence  in  Discourse 
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Selfridge  Inference  and  Learning  in  C  Computer  Model  — 

Salveter  Inferring  Building  Blocks  for  Knowledge  Representation 
Charniak  Context  Recognition  in  Language  Comprehension 
Schank  Reminding  and  Memory  Organization  MOPs 

— DB - DATA  BASE  RETRIEVAL/REPRESENTATION 


Fred  M.  Hall,  Economic  Analysis  for  Data  Base  Management,  Proceedings  Annual 
Reliability  and  Maintainability  Symposium.  1982,  343-352,  Evaluation  Research  Corp. 
Cost-effective  field  data-  collection  program  to  support  a  system  readiness  improvement 
program.  Each  of  the  services  currently  processes  a  million  records  per  month  but  not 
the  right  kind  of  R&M  data. 

Special  Issue  on  AI  and  Database  Research,  SIGART  Newsletter,  October  1983,  32-72 
There  are  more  than  30  half-  to  full-page  summaries  of  research  in  AI  and  Database 
research.  Included  information  on  rules,  RESEDA  (bibliographic  data),  VIEW  and 
TOPOGRAPHIC  (graphical),  natural  language  interfaces,  Information  Management  at  ISI, 
and  other  direct  AI  type  of  databases. 


— ES - - - . -  EXPERT  SYSTEMS 


Avron  Barr,  Edward  A.  Feigenbaum,  The  Handbook  of  Artificial  Intelligence,  1982, 
William  Kaufmann  Inc,  3  Volumes 

B.  G.  Buchanan,  New  Research  on  Expert  Systems,  269-300,  Machine  Intelligence  10,  3. 

E.  Hayes,  Donald  Michie,  University  of  Edinburgh,  Wiley,  1982 

Broad  based  summary  of  the  past  work,  state  of  the  art,  and  near  term  problems  in  AI. 
Deals  with  representation  and  control,  explanation,  knowledge  acquisition,  validation, 
experimentation,  and  choosing  a  framework.  Long  appendix  is  example  of  MYCIN.  Many 
references. 

National  Bureau  of  Standards,  An  Overview  of  Expert  Systems,  May  1982,  PB82-227547, 
NBS,  74,  Best  overview  for  1982,  ESI 

W.  B.  Gevarter,  Expert  Systems:  Limited  but  Powerful,  IEEE  Spectrum  August  1983,  39- 
45,  most  uptodate  and  consise  information  on  E.S. 

Dana  S.  Nau,  Expert  Computer  Systems,  Computer,  IEEE,  February  1983,  63-85, 

Excellent  technical  overview 

F.  Hayes-Roth,  D.A.  Waterman,  D.B.  Lenat,  Building  Expert  Systems,  Addison-Wesley 
ISBN  0-201-10686-8,  Teknowledge  Series  Vol  1,  400  pg. 

William  R.  Swartout,  Explaining  and  Justifying  Expert  Consulting  Programs,  I3AI81  815- 
823,  MIT 

XPLAIN  explains  to  the  user  what  it  does  and  why  by  AUTOMATIC  PROGRAMMING 

Jeffrey  A  Barnett,  Some  Issues  of  Control  in  Expert  Systems.  Proceedings  on  Cyberne¬ 
tics  and  Society,  IEEE,  1982.  l-5,U5C/Information  Sciences  Institute  Marina  del  Rey 
Cal. 
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Wants  AI  systems  to  fail  softly  at  the  boundaries  of  their  competence.  The  sort 
reasoning  chains  should  have  the  ability  to  fall  back  on  general  principles.  Ths  large 
grained  knowledge  comes  from  expert  rules  while  the  smaller  chunks  of  knowledge  will 
come  from  causal  rules. 

Rodger  T.  Hartley,  How  Expert  Should  an  Expert  System  Be,  I3AI-81,  862-867 

Motoi  Suna,  A  Carlisle  Scott,  E.H.  Shorliffe,  An  Approach  to  Verifying  Completeness  and 
Consistancy  in  a  Rule-Based  Expert  System,  Dept  of  Computer  Science,  Stanford,  STAN- 
CS-82-922,  June  1982,  also  AAAI-82? 

G.  E.  Kopec,  A.  V.  Oppenheim,  R.  Davis,  Knowledge-Based  Signal  Processing,  Trends  & 
Perspect.  Signal  Process  (USA),  Vol.  2,  no.  3,  1-6,  July  1982 

Discusses  'knowledge-based  signal  processing'— a  term  used  to  describe  systems  that 
tightly  integrate  artificial  intelligence  (AI)  and  signal  processing  which  attempts  to 
combine  techniques  from  the  two  disciplines  more  imaginatively  than  in  the  past. 
Researchers  in  the  signal  processing  community  are  increasingly  becoming  aware  that 
combining  the  architecture  and  methodology  of  AI  with  more  traditional  tools  and 
techniques  can  lead  to  significant  advantages. 

K.  L.  Clark,  F.  G.  McCabe,  PROLOG:  A  Language  for  Implementing  Expert  Systems, 

455-977,  Machine  Intelligence  10,  J.  E.  Hayes,  Donald  Michie,  University  of  Edinburgh, 
Wiley,  1982 

J.  McDermott,  XSEL:  A  Computer  Sales  Person's  Assistant,  Machine  Intelligence  10,  J. 
E.  Hayes,  Donald  Michie,  University  of  Edinburgh,  DA  FD,  Wiley,  1982,  325-337 
New  version  of  DEC  R1  program  for  configuring  VAX  computers. 

C.  Freska  Linguistic  Description  of  Human  Judgments  in  Expert  System  and  in  the  Soft 
Sciences  Proceedings  of  the  Twenty -Sixth  Annual  Meeting  of  the  Sonety  for  General 
Systems  Research,  Jan.  1982,  429-434 

Ruven  Brooks,  A  Comparison  Among  Four  Packages  for  Knowledge-based  Systems, 
University  of  Texas  Medical  Branch,  15-18  June  1982,  279-283,  1981,  IEEE 
Four  tools  for  the  construction  of  knowledge-based  systems,  EMYCIN,  EXPERT,  AIMDS, 
and  the  blackboard  option  of  AGE,  are  compared  in  regard  to  their  capabilities  for 
representing  three  kinds  of  knowledge:  judgemental  or  decision-making  knowledge, 
explanatory  knowledge,  and  modeling  knowledge. 

Mark  Stefik,  Jan  Alkins,  Robert  Balzer,  John  Benoit,  L.  Birnbaum,  Frederick  Hayes- 
Roth,  Earl  Sacerdoti,  The  Organization  of  Expert  Systems*,  A  Tutorial,  Xerox  Palo  Alto 
Research  Center,  Artificial  Intelligence,  March  1982,  V.  18  #2,  135-173 

Mordechai  Halpern,  A  Case  Study  of  a  Flexible  Automated  Knowledge  System,  IEEE  1982 
Cybernetics  and  Society,  28-30  October  1982,  613-617 

We  represent  a  methodology  for  the  development  of  a  class  of  knowledge  systems  that 
are  capable  of  responding  to  unanticipated  queries.  This  methodology  is  demonstrated 
through  the  detailed  construction  of  a  prototype  case  study. 

Avron  Barr,  Meta-Knowledge  and  Cognition,  6th  IJCAI  1979  Tokyo  Japan,  20-23  August 
1979,  31-33 

In  AI  knowledge  representation  schemes,  structures  that  describe  other  structures  are 
said  to  represent  "meta-knowledge”,  or  knowledge  about  other  knowledge.  After 
describing  some  studies  of  human  behavior  that  demonstrate  people's  ability  to  reason 


about  what  they  know  and  about  how  they  reason,  we  review  the  use  of  explicit  meta¬ 
knowledge  in  several  recent  AI  systems.  The  concept  of  meta-level  knowledge  captures 
intrinsic,  commonplace  properties  of  human  cognition  that  are  central  to  an  under¬ 
standing  of  knowledge  and  intelligence. 

Robert  Blum,  Discovery,  Confirmation,  and  Incorporation  of  Casual  Relationships  from  a 
Large  Time-Oriented  Clinical  Data  Base:  The  RX  Project,  Computers  and  Biomedical 
Research.  15, //2,  April  1982,  164-187 

AI  Consultation  Techniques  in  Systems  Support,  John  K.  Scully,  AUTOTESTCON 
Proceedings,  1983,  324-330,  IEEE, 
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MAINTENANCE  EXPERT 


Rodger  T.  Hartley,  CRIB:  Computer  Fault-Finding  Through  Knowledge  Engineering, 
Computer,  March  1984,  page  76-83. 

Excellent  review  of  a  proven  system  which  learns  diagnostics. 

T.R.  Addis,  Expert  Systems:  An  Evolution  in  Information  Retrieval,  Information 
Technology;  Research  and  Development,  1982,  pg  301-324,  includes  CRIB  maintenance 
expert  description. 

Joseph  G.  Wohl,  Maintainability  Prediction  Revisited:  Diagnostic  Behavior  System 
Complexity,  and  Repair  Time,  Transactions  on  Systems.  Man,  and  Cybernetics,  IEEE, 
May /June  1982,  241-250 

The  author  is  able  to  predict  repair  time  very  accurately  in  two  categories,  under  two 
hours  and  over  two  hours.  Sixty-five  to  eighty  percent  of  the  repairs  were  less  than  two 
hours  when  there  was  BIT,  self -test,  self -diagnostic,  a  good  maintenance  manual,  or 
good  equipment  packaging  and  modularization.  See  also  his  1982  paper  on  Cognitive 
Capability  vs  System  Complexity  in  Electronic  Maintenance,  Proceedings  on  Cyber  and 
Society  1982 

Stephen  R.  Olson,  Bruce  A.  McGraw,  Stephen  H.  Morriss,  Education,  Training,  and 
Maintenance  Aiding:  New  Concepts  and  Solutions,  May  1981,  NAECON  1981,  IEEE.  1382- 
1389 

This  paper  summarizes  the  potential  of  CAI,  CMI,  and  maintenance  aiding  and  how  such 
concepts  can  be  introduced  as  a  total  system.  The  paper  also  addresses  requirements 
appropriate  to  the  delivery  device  itself  and  includes  discussion  of  speech  synthesis, 
graphics,  and  material  development. 

Naval  Research  Advisory  Committee  Report  on  Reliability,  Availability,  and  Maintaina¬ 
bility  (RAM)  Reliability,  Availability  and  Maintainability  (RAM),  Naval,  Research 
Advisory  Committee  report  on;  NRAC  82-2,  October  1982,  Findings  of  committee 
include: 

1)  Lack  of  reliability  is  seen  as  lack  of  availability. 

2)  M,  FI,  BIT,  and  redundancy  are  key  factors  influencing  A. 

3)  Good  front-end  engineering  is  the  most  effective  key  step  toward 
improving  RAM. 

4)  Many  RAM  problems  cannot  be  anticipated  and  will  show  up  during  FSD 
testing,  during  transition  to  production,  during  production,  and  after  field 
introduction. 

The  only  way  to  reach  achievable  levels  of  R  is  by  an  iterative  method  of 
testing,  identifying  etc. 
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5)  Panel  could  not  quantify  the  cost  and  benefits  of  achieving  increased 
RAM.  The  formal  R  program  costs  4-6%  of  the  RDT&E  costs  and 
additional  costs  of  maybe  10-20%  of  the  RDT&E  costs  if  iterative 
redesign  and  test  are  required  during  development  and  transition  to 
production  wag). 

R&M  Study  of  IDA  Vol  in,  August  1983,  Institute  of  Defense  Analysis,  HIGH  PAY-OFF 
AREAS  Vol  III  part  7.0 

"CAD  can  truly  make  R&M  part  of  mainstream  design  engineering."  (pg  4) 

"A  maturation  program  for  BIT  has  also  proven  to  be  essential"  (pg  5) 

SIGNIFICANT  FINDINGS/OBSERVATIONS  part  8.0 

"In  7  of  the  8  cases  studied,  financial  incentives  were  judged  to  have  been  a  significant 
factor  in  focusing  management's  attention  on  R&M"  (pg  1.) 

"  It  seems  though  that  R&M  emphasis  may  have  been  lacking  during  the  concept  and  even 
in  development  and  validation  where  the  potential  of  a  system  is  really  established" 
(pg  2.) 

"Accuracy  and  availability  of  field  data  in  most  cases  was  judged  to  be  insufficient  to 
support  requirements  development,  short  term  trend  analysis,  and  timely  problem 
identification"  (pg  4.) 

"A  reliability  growth  program  should  use  data  from  all  test  and  operational  sources.  The 
program  should  be  structured  at  the  beginning  to  treat  every  failure  or  operating 
anomaly  as  a  potential  reliability  growth  opportunity"  (pg  5.0-26) 

"..strong  consideration  should  be  given  to  extending  planned  BIT  maturation  into  the 
operational  phase."  (pg  5.0-29) 

Randall  Davis,  Howard  Shrobe,  Representing  Structure  and  Behavior  of  Digital  Hard¬ 
ware,  75-82,  Computer,  Oct.  1983,  MIT  AI  LAB. 

Harold  S.  Balaban,  William  R.  Simpson,  Testability/Fault  Isolation  by  Adaptive  Strategy, 
Proceedings  Reliability  and  Maintainability  Symposium  1983,  344-349,  ARINC 
Develops  nodal  connection  interaction  analysis  on  an  Apple  computer.  Program  is  called 
STAMP. 

W.  R.  Simpson,  3.  R.  Agre,  Adaptive  Fault  Isolation  with  Learning,  IEEE 
AUTOTESTCON,  1983,  331-335,  ARINC 

Directly  incorporates  field  information  of  test  time  and  failure  frequency  to  optimize 
the  testing  sequence.  Does  not  currently  provide  for  additional  test  strategies  to  be 
recorded  or  suggested  by  the  test  operator.  Tests  can  be  optimized  for  cost  (peace)  or 
time  (war)  by  the  operator  at  test  time.  Explanations  can  be  requested  from  the  system, 
such  as  video  images  of  how  to  perform  the  test  (currently  on  videotape).  Presently  all 
test  strategies  are  based  on  all  tests  at  a  particular  location.  This  could  be  changed  to  a 
weighted  moving  average  at  a  location  or  be  based  on  information  from  more  than  one 


Shunichi  Toida,  A  Graph  Model  for  Fault  Diagnosis,  Journal  of  Digital  Systems,  VI  ,//4. 
1982,  345-365, 

Methods  to  determine  the  smallest  number  of  test  points  needed,  with  such  constraints 
as  mimimizing  the  total  number  of  tests. 

L.  F.  Pau,  Failures  Detection  Processes  by  Pattern  Recognition  and  Expert  Systems, 
submitted  for  publication  by  Pattern  Recognition  Letters.  Includes  meta -rules  for 
electronic  diagnosis,  the  most  advanced  in  world. 


L.  F.  Pau,  3.  Rasmusson,  Application  of  Pattern  Recognition  to  Failure  Analysis  and 
Diagnosis,  429-446,  Human  Detection  and  Diagnosis  of  System  Failures.  1981 

L.  F.  Pau,  An  Adaptive  Signal  Classification  Procedure  Application  to  Aircraft  Engine 
Monitoring,  Pattern  Recognition.  Vol  9,  1977,  121-130 

L.  F.  Pau,  Failure  Diagnosis  and  Performance  Monitoring,  Marcel  Dekker  NY  $49  1981, 
427,  Battelle  Switzerland, 

This  is  the  only  book  which  clearly  addresses  the  theoretical  and  practical  aspects  of 
sophisticated  techniques  for  failure  analysis.  He  discusses  the  concepts  of  which  we  call 
self-improving  diagnostics  (learning),  Smart  Built-in-Test  (observability)  without 
referring  to  AI  once(  pattern  recognition).  Some  of  the  chapter  titles  are:  Degradation 
Processes  (for  mechanical  systems),  Test  sequencing  with  a  priori  information  for  failure 
locatization,  R&M  data  banks,  Failure  and  Maintenance  data  analysis,  Correspondence 
Analysis  for  Failure  analysis,  Automated  Failure  diagnosis  by  Pattern  recognition, 
adaptive  failure  diagnosis  and  performance  monitoring. 

Michael  L.  Arington,  The  'Experience  Algorithm'  in  Automating  Isolation  of  Single  and 
Multiple  Faults,  ATE  Central  Proceedings,  October  1983,  Magnavox  Ft.  Wayne  Indiana 
Describes  some  concepts  for  what  others  call  self  improving  diagnostics.  Decision 
program  would  choose  the  most  likely  failures  based  on  the  highest  number  of  previous 
failures.  System  would  also  compare  fault  signatures.  "Experience  programs  are  not 
meant  to  replace  or  even  reduce  fault  isolation  algorithms  or  the  need  for  testability  but 
they  can  enhance  and  complement  them  technically  and  economically."  "Some  cases  will 
require  as  little  as  tabulation  tables  of  correct  and  incorrect  diagnosis.  Other  applica¬ 
tions  may  require  the  sophistication  of  AI". 

LES:  A  Model-Based  Expert  System  for  Electronic  Maintenance,  T.J.  Laffey,  W.  A. 
Perkins,  O.  Firschein,  Fall  1983,  Lockheed  Palo  Alto  Research  Lab 

The  Lockheed  Expert  System  (LES)  uses  expert  knowledge  in  the  form  of  IF-THEN  rules 
and  knowledge  of  the  structure,  function,  and  causal  relations  of  the  device  under  test.  It 
graphically  displays  the  interrelationships  of  the  device  being  tested.  This  aids  in  the 
technician's  diagnosis  process.  It  has  been  applied  to  a  large  electronic  system  of  16 
equipment  cabinets  which  included  built-in-test.  Future  systems  should  integrate  the 
BITE  with  LES,  so  as  to  eliminate  much  of  the  human  interaction.  They  are  currently 
adding  voice  input/output  to  free  themselves  from  the  terminal  console  (like  TI  PEAM). 
They  also  want  the  system  to  adapt  to  the  skill  of  the  user,  but  are  not  yet  concerned 
with  self -improving  diagnosis. 

George  C.  Sumner,  Knowledge-Based  Systems  Maintenance  Applications,  IEEE  1982 
AUTOTESTCON,  Dayton,  Ohio,  472-473 

Knowledge  based  systems  promise  improvement  in  failure  diagnosis  and  can-not- 
duplicate  conditions  of  electronics  maintenance. 

D.  R.  Antonelli,  The  Application  of  Artificial  Intelligence  To  A  Maintenance  and 
Diagnostic  Information  System  (MDIS),  1983,  Proceedings  of  the  Joint  Services 
Workshop  On  Artificial  Intelligence  and  Maintenance.  Boulder,  Co. 

To  be  published 

Artificial  Intelligence  to  Support  Avionics  Maintenance  Diagnostics  A  State-of-the-Art- 
Assessment,  Dr.  Curt  Biais,  Dr.  Benjamin  Schwartz,  Mr.  Richard  Schwarz,  Dr.  Stephen 
Tolchin,  Dr.  Chelsea  White,  System  Exploration  Inc.,  Jan  1984,  for  AF  Human  Resources 
lab,  Wright  Patterson.  No  publication  number. 


Causal  and  Teleological  Reasoning  In  Circuit  Recognition,  Johan  deKieer,  Sept 

1979, 1-208 

This  thesis  presents  a  theory  of  human-like  reasoning  in  the  general  domain  of  designed 
physical  systems,  and  in  particular,  electronic  circuits.  One  aspect  of  the  theory,  causal 
analysis,  describes  how  the  behavior  of  individual  components  can  be  combined  to  explain 
the  behavior  of  composite  systems.  Another  aspect  of  the  theory,  teleological  analysis, 
describes  how  the  notion  that  the  system  has  a  purpose  can  be  used  to  aid  this  causal 
analysis. 

J.  W.  Lewis,  PhD,  F.  S.  Lynch,  ScD,  GETREE:  A  Knowledge  Management  Tool,  1983, 
IEEE,  CHI 

General  Electric  Company,  Corporate  Research  and  Development.  In  practical  expert 
systems,  regardless  of  inference  technology,  the  accuracy  and  completeness  of  the 
knowledge  base  determine  expert  system  performance,  and  the  cost  of  acquiring  that 
knowledge  base  tends  to  dominate  all  other  hardware  and  software  costs.  To  reduce 
knowledge  acquisition  cost  and  error  rate,  GETREE— a  new  interactive  knowledge 
management  system— is  being  designed  and  implemented  in  GE  Corporate  Research.  In 
the  new  system,  die  knowledge  base  is  represented  as  an  (almost  tree-like)  network  of 
nodes  .which  can  be  displayed  and  manipulated  on  a  personal  computer  workstation. 
Users—even  unsophisticated  users— can  build  and  navigate  these  networks,  modify  nodes 
and  connecting  arcs,  verify  correctness  visually,  follow  the  execution  of  inference 
engines,  and  generate  equivalent  code  to  be  run  in  more  constrained  target  environ¬ 
ments.  Consequently,  with  GETREE  the  user-organization  can  have  full  responsibility 
for  and  control  of  the  knowledge  base. 


— R 


ROBOTICS  — 


Irene  C.  Peden,  Artificial  Intelligence  and  Robotics,  Army  Science  Board,  Sept.  1982 


H.  Mcllvaine  Parsons,  Greg  P.  Kearsley,  Robotics  and  Human  Factors:  Current  Status 
and  Future  Prospects,  Human  Factors.  24,  5,  1982,  535-552 

The  most  important  question  is  the  division  of  labor  between  robots  and  humans.  Humans 
participate  in  robot  systems  in  9  activites:  surveillance,  intervention,  maintenance, 
hardware  and  software  design  of  interfaces  with  operators,  procedure  development, 
accident  prevention,  and  training. 

N.  S.  Rajaram,  Knowledge  Based  Systems  for  Intelligent  Robotics,  IEEE _ 

Needs  of  the  space  program.  Useful  methods  may  be  theorem  proving  and  predicate 
calculus. 

A  Computer  Vision  and  Robotics  Laboratory,  R.  A.  Jarvis,  IEEE  Computer,  June  1982,  9- 
23 

Color  vision  is  combined  with  a  robot  arm  and  ultra-sonic  ranging  in  this  off-the-shelf 
system.  Color  and  range  aids  in  scene  segmentation  with  additional  feedback  via  the 
robot  arm. 

M.  Brady,  J  M.  Hollerbach,  T.  L.  Johnson,  T.  Lozano-Perez,  M  Mason,  Robot  Motion: 
Planning  and  Control,  McGraw  Hill  $37.50,  MIT  AI  Lab 
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— AP - AUTOMATIC  PROGRAMMING 


Francesco  Caio,  Giovanni  Guida,  Marco  Somaivico,  Problem  Solving  As  A  Basis  for 
Program  Synthesis:  Design  and  Experimentation  of  the  BIS  System,  Int.  Jrnl.  Man- 
Machine  Studies.  1982,  pg  173-188,  automatic  programming  synthesis,  DRA 
The  paper  is  devoted  to  assessing  and  motivating  the  central  role  that  classical  problem 
solving  techniques  can  play  in  the  development  of  a  program  synthesis  system.  Uses 
bidirectional  synthesis,  which  considers  the  task  of  automatic  programming  as  based  on 
the  cooperation  between: 

1)  a  top-down  activity  of  problem  reduction 

2)  a  bottom-up  activity  of  binding  together  of  program  modules  in  a  structured 
way 

Moreover,  this  approach  is  experimentally  supported  by  the  development  of  the  BIS 
(Bidirectional  Synthesizer)  system  for  interactive  synthesis  of  LISP  Programs.  BIS  is 
developed  on  a  UNIVAC  1100/80  and  is  written  in  LISP. 


^v,vTy.y:\ 
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Richard  C.  Waters,  A  Method  for  Automatically  Analyzing  Programs,  6th  I3CAI,  20-23 
August  1979,  pg  935-941. 

This  paper  presents  a  method  for  automatically  analyzing  programs  and  discusses  why  it 
is  a  useful  way  to  look  at  programs.  The  method  is  based  on  the  idea  that  there  are  only 
a  few  basic  ways  in  which  the  logical  structure  of  programs  is  built  up.  An  experiment  is 
presented  which  shows  that  this  accounts  for  the  structure  of  a  large  class  of  programs. 
The  paper  discusses  how  the  method  can  be  used  to  automatically  analyze  the  structure 
of  a  program,  and  how  the  resulting  analysis  can  be  used  to  guide  a  proof  of  correctness 
for  the  program.  An  automatic  system  is  described  which  performs  this  type  of  analysis. 


t.  • 


-VI - VISION 


W.  A.  Perkins,  Model-Based  Component  Board  Inspection,  1982,  IEEE  6th  International 
Conf.  on  Pattern  Recognition 

Gray-scale  computer  vision  for  inspection  of  a  flat  surface:  looking  for  correct 
component,  correct  orientation  and  position,  and  other  features. 

D.H.  Ballard,  C.M.  Brown,  Computer  Vision,  1982,  523,  Prentice-Hall,  Strong  AI  flavor 
to  the  book, 


*  ►"**  u’V  k*“  , 


George  Koutures  "Automated  Optical  Inspection  Paces  Future  PC  Production", 
Electronic  Packaging  and  Production  pg.  126-128,  June  1983 


»  • 


-P - PATTERN  RECOGNITION 


Yoh-Han  Pao,  George  W.  Ernst,  Context-Directed  Pattern  Recognition  and  Machine 
Intelligence  Techniques  for  Information  Processing  :  a  Tutorial,  IEEE  EH0193-3,  1982, 
577+, 

This  is  a  collection  of  papers,  primarily  during  last  5  years 
R  Nevatia,  Machine  Perception,  1982,  Prentice  Hall,  209  pages. 
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Charles  L.  Forgy  RETE:  A  Fast  Algorithm  for  the  Many  Pattem/Many  Object  Pattern 
Match  Problem,  Artificial  Intelligence.  September  1982,  17-37 

By  compiling  objects  and  patterns  into  single  word  descriptions,  RETE  is  able  to  very 
quickly  perform  searches  for  many  objects  with  many  patterns  in  a  single  scan. 

Douglas  S.  Tudhope,  John  V.  Oldfield,  A  High-Level  Recognizer  for  Schematic  Diagrams, 
IEEE  Transactions  on  Computer  Graphics  and  Applications,  May/ June  1983,  33-40, 


Uses  AI  to  store  circuit  diagram  information  in  a  global,  modifiable  database. 


—CAD - COMPUTER  AIDED  DESIGN 


G.J  Sussman,  Computer  Aided  Evolutionary  Design  for  Digital  Integrated  Systems,  MIT 

AIM  526,  May  1979,  AD-A078124 
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Mark  J  Stefik,  Johan  de  Kleer,  Prospects  for  Expert  Systems  in  CAD,  Xerox  Palo  Alto 
Research  Center,  Computer  Design,  April  21  1983,  65-76, 

PALLADIO  is  an  expert  system  developed  by  Xerox  and  Stanford  for  circuit  design.  It 
uses  Linked  Module  Abstraction  (LMA),  CRL  and  CPS. 

M.  W.  Krueger,  R.  E.  Cullingford,  D.  A.  Beliavance,  Control  Issues  in  a  Multiprocess 
Computer-Aided  Design  System  Containing  Expert  Knowledge,  IEEE  1981  Cybernetics 
and  Society,  139-143 

CADHELP  is  a  knowledge-based  CAD  system  for  the  design  of  digital  logic  circuits.  This 
system  contains  a  knowledge  representation  describing  each  of  the  graphic  features  that 
the  system  provides.  These  feature  scripts  can  be  used  to  generate  graphic  demonstra¬ 
tion  and  text  explanations  of  the  use  of  these  features. 

The  use  of  scripts  to  document  the  system  has  been  extended  to  include  control  scripts. 
These  describe  the  interaction  of  system  experts  responsible  for  staging  graphic  modules 
and  assimilation  of  their  results  into  a  Design  Knowledge  Structure.  Also  of  interest  is 
the  fact  that  the  control  scripts  are  used  to  control  the  events  they  describe.  Thus, 
there  is  no  possible  conflict  between  execution  and  documentation. 

Tan  E.  Kelly,  Louis  L.  Steinberg,  Rutgers  University,  The  Critter  System:  Analyzing 
Digital  Circuits  by  Propagating  Behaviors  and  Specifications  AAAI-82,  284-289,  Reasons 
about  digital  HW  designs  but  needs  to  be  extended  to  practical  circuits  which  include 
feedback  and  components  with  state. 

Tom  M  Mitchell,  Louis  I  Steinberg  etc,  An  Intelligent  Aid  for  Circuit  Redesign,  AAAI 
1983,  Rutgers  University 

Knowledge  based  circuit  redesign  has  causal  reasoning  about  circuits  and  purpose  of  the 
circuit.  This  redirection  of  their  CRITTER  work  has  a  narrow  scope,  and  appears  to  have 
some  near-term  uses.  This  has  some  applicability  to  the  testability  assistant  which  must 
also  compare  effects  of  changes  to  a  circuit. 

Harold  Brown,  Christopher  Tong,  Gordon  Foyster,  Palladio:  An  Exploratory  Environment 
for  Circuit  Design,  Computer.  December  1983,  Stanford 
VLSI  prototype  AI  circuit  designer. 

Thomas  L.  Fennell,  Thomas  A.  Nicolino,  Computer  Aided  Testability,  1984  Proceedings 
Annual  Reliability  and  Maintainability  Symp..  RADC  contract  to  Boeing, 

Condensed  information  of  RADC  TR-83-25/,  outlined  in  section  8.4.4  of  this  report. 
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T.  3.  Kowalski,  D.C.  Thomas,  The  VLSI  Design  Automation  AssistantsLeaming  to  Walt 
IEEE  International  Symposium  On  Circuits  and  Systems,  May  1983,  186-189,  Carnegie- 
Mellort  University 

A  Design  Automation  Assistant  using  AI  techniques  was  used  to  design  different  versions 
of  the  6502  microprocessor.  Use  of  OPS-5  started  with  70  rules  and  grew  to  130  rules. 
Future  Design  Automation  Assistants  are  planned  to  include  design  of  a  VAX  computer. 

C.3.  Rimkus,  M.  R.  Wayne,  D.  D.  Cheng,  F.  3.  Magistro,  Structured  Design  Verification: 
Function  and  Timing,  20th  Design  Automation  Conference,  1983,  246-251,  IBM- 
Poughkeepsie 

VLSI  needs  to  be  100%  error  free  the  first  time.  Table  gives  change  cost  factors  at 
various  points  in  the  design  cycle.  A  change  in  engineering  objectives  has  a  cost  factor 
=  1.  A  change  at  detailed  logic  mode  has  a  cost  of  20,  but  product  rework  has  cost  factor 
of  possibly  10,000. 

Max  Schindler,  Computer-Aided  Engineering  Comes  of  Age,  Electronic  Design,  Nov.  1 1 , 
1982,97-104 

Most  are  based  on  68000.  With  CAE,  the  designer  creates  a  circuit  (or  machine)  in 
computer-readable  form  from  the  very  beginning.  The  same  data  base  can  then  provide 
the  inputs  either  for  a  placement  or  routing  or  for  a  simulation  routine;  if  the  results 
lead  to  design  changes,  the  inputs  are  automatically  integrated  into  the  data  base.  CAE 
is  done  on  a  workstation,  CAD  can  be  done  in  batch. 

A.  L.  Elias,  Computer-Aided  Engineering:  The  AI  Connection,  Astronautics  and 
Aeronautics,  3uly/August  1983,  48-54,  MIT 

Martin  F.  Rooney,  Stephen  E.  Smith,  Artificial  Intelligence  in  Engineering  Design, 
Computers  and  Structures,  1983,  16,  279-288 

Richard  E.  Cullingford,  Myron  W.  Kreuger,  Mallory  Selfridge,  Marie  A.  Bienkowski, 
Automated  Explanations  as  a  Component  of  a  Computer-Aided  Design  System,  IEEE 
Transactions  on  Systems,  Man,  and  Cybernetics,  March/April  1982,  SMC-12,  2,  168-181, 
Univ  of  Connecticut,  Rensselaer  Polytechnic  Describes  CADHELP 

T.  3.  Kowalski,  D.E.  Thomas,  The  VLSI  Design  Automation  Assistant:  First  Steps,  IEEE 
CH  1856,  1983,  126-129 

Henry  L.  Nattrass,  Glen  K.  Okita,  Some  Computer  Aided  Engineering  System  Design 
Principles,  20th  Design  Automation  Conference,  1983,  571-577,  Hewlett  Packard 
Describes  CAE  at  HP,  including  Hierarchical  design  file,  database, etc  similar  to 
commercial  engineering  work  stations. 


Gil  Bassal,  Designing  Computers  —  Special  Report  Electronic  Design,  April  19,  1984,  91- 
104 

Good  overview  of  engineering  workstation  projects  of  ten  companies.  Includes  ATPG  and 
other  simulation  packages.  LISP  Machines'  Newdraw  logic  design  program  includes  some 
AI  features 


Thomas  H.  Bruggere,  Elizabeth  Holloman,  "Tools  for  Computer-Aided  Engineering",  IEEE 
Computer  Graphics  and  Applications,  Dec  1983,  pg  48-53. 

Explains  electronic  design  tools  and  other  tools  for  the  Mentor  Graphics  engineering 
work  station. 
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P.B.  Stevens,  Workstations  and  Special  Purpose  Hardware,  Database  Engineering,  March 
1984. 

Stephen  Evanchzifc,  Special  Report:  York  Stations,  Integrating  the  Engineers  Environ¬ 
ment,  Electronics.  May  1984,  pg  121-130. 

Includes  price  trends.  Mentor  has  24%  market  share,  Daisy  32%. 


— MT - MAINTENANCE  TRAINING 


Ruston  M.  Hunt,  Richard  L.  Henneman,  William  B.  Rouse,  Characterizing  the  Develop¬ 
ment  of  Human  Expertise  in  Simulated  Fault  Diagnosis  Tasks  IEEE  1981  Cybernetics  and 
Society,  369-374, 

Discusses  a  wide  range  of  human  problem  solving  characteristics  useful  in  predicting  and 
evaluating  performance  on  simulated  fault  diagnosis  tasks.  Performance  metrics 
considered  include  project  measures  such  as  error  frequency  and  information  gain  per 
action.  These  measures  are  used  to  explain  the  varying  levels  of  expertise  exhibited  by 
subjects  on  the  simulated  tasks. 

P.  E.  Van  Hemel,  W.  3.  King,  Simulation  Techniques  In  Operator  And  Maintenance 
Training,  Performance  Assessment,  And  Personnel  Selection,  Computer  and  Indus. 
Engng..  5,2,105-112, 

Examines  the  use  of  synthetic  simulators  for  training  personnel  who  operate  and 
maintain  electronic  and  other  systems.  It  uses  past  problems  as  a  basis  for  simulation. 
Has  examples  of  Army,  Navy  and  Air  Force  programs. 

D  Sleeman,  3.  S.  Brown,  Intelligent  Tutoring  Systems,  Academic  Press,  1982 

Francis  D.  Fisher,  Computer-Assisted  Education:  What's  Not  Happening,  3oumal  of 
Computer-Based  Instruction.  Summer  1982,  9  #1, 

Dr.  3anathin  Miller,  Chairperson,  The  Status  of  Training  Technology  in  the  ATE 
Community,  IEEE  1982  AUTOTESTCON,  12-14  October  1982,  110-113 
Outlines  future  changes  in  ATE  technical  training  programs  due  to  advances  in 
instructional  media  including:  interactive  video,  videodisc,  CAD/CAM,  computer 
assisted/managed  instruction,  distributed  information  systems,  robotics,  etc. 


-—ML - MACHINE  LEARNING 


Ryszard  S.  Michalski,  3aime  G.  Carbonell,  Tom  M.  Mitchell,  Machine  Learning  :  An 
Artificial  Intelligence  Approach,  Tioga  Publishing  Co.,  1983,  57 2, 

Very  comprehensive  coverage.  This  book  appears  to  be  a  first  for  this  field. 

SUB3ECTS:  Learning  from  Examples  Inductive  Learning 

In  Problem-Solving  and  Planning  By  Analogy  By  Experiment 
Learning  from  Observation  and  Discovery  AM  EURISKO  BACON 
Conceptual  Clustering 

Learning  from  Instruction  Advice  transformation  into  Search  Proc. 
Learning  by  Being  Told  KLAUS  NONOKLAUS 
Applied  Learning  Systems  Chess  CAI 
Also  has  a  bibliography  of  572  items 


Jack  Mostow,  Machine  Learning  Workshop,  SIGART  Newsletter,  October  1983,  24-31 
Provides  a  summary  and  comments  on  the  papers  and  talks  presented  at  the  1983 
Machine  Learning  Workshop. 

Proceedings  of  the  International  Machine  Learning  Workshop,  Dept,  of  Computer 
Science,  U.  of  Illinois,  Urbana,  June  22-24  1983,  234, 

R.S.  Michalski,  R.L.  Chilausky,  Knowledge  Acquisition  By  Encoding  Expert  Rules  Versus 
Computer  Induction  From  Examples,  lnt.  Jml.  Man-Machine  Studies.  16,  1980,  63-87, 
Current  methods  of  knowledge  acquisition  rely  entirely  on  the  direct  representation  of 
knowledge  of  experts,  which  usually  is  a  very  time-  and  effort-consuming  task.  The 
paper  presents  results  from  an  experiment  to  compare  the  above  method  of  knowledge 
acquisition  with  a  method  based  on  inductive  learning  from  examples.  The  comparison 
was  done  in  the  context  of  developing  rules  for  soybean  disease  diagnosis  and  has 
demonstrated  an  advantage  of  the  inductively  derived  rules  in  performing  a  testing  task 
(which  involved  diagnosing  a  few  hundred  cases  of  soybean  diseases). 

R.  S.  Michalski,  R.  L.  Chilausky,  Learning  by  Being  Told  and  Learning  from  Examples:  An 
Experimental  Comparison  of  the  Two  Methods  of  Knowledge  Acquisition  in  the  Context 
of  Developing  an  Expert  System  for  Soybean  Desease  Diagnosis,  int.  j.  policy  Anal,  and 
Inf.  Systems.  4,  2,  125-161,  June  1980 

This  paper  contrasts  the  method  of  knowledge  acquisition  by  encoding  decision  rules  of 
human  experts  with  that  of  learning  the  rules  (by  means  of  an  inductive  program)  from 
examples  of  decisions  made  by  these  experts. 

Ryszard  S.  Michalski,  Robert  E.  Stepp,  Automated  Construction  of  Classification: 
Conceptual  Clustering  Versus  Numerical  Taxonomy,  IEEE  Transactions  on  Pattern 
Analysis  and  Machine  Intelligence.  July  1983,  396-410,  Univ.  of  111. 

His  CLUSTER/2  program  is  compared  to  18  numerical  taxonomy  methods  for  two 
example  cases. 

Dana  Nau,  James  Reggia,  Pearl  Wang,  Knowledge-based  Problem  Solving  without 
Production  Rules,  Trends  and  Applications:  Automating  Intelligent  Behavior.  May  1983, 
IEEE  Cat  83CH1887-9. 

Generalization  of  set  covering  problem  which  captures  several  intuitively  plausible 
features  of  human  diagnostic  interference  and  directly  address  problem  of  multiple 
simultaneous  disorders.  KMS.HT  is  a  Knowledge  Management  diagnostic  subsystem 
which  can  "hypothesize  and  test". 

John  E.  Gaffney,  Jr.,  I.  Randy  Racer,  A  Learning  Interpretive  Decision  Algorithm  For 
Severe  Storm  Forecasting  Support,  Trends  and  Applications:  Automating  Intelligent 
Behavior.  Applications  and  Frontiers,  Proceedings,  May  1983,  IEEE  Cat  83CH1887-9. 
National  Weather  Service  Silver  Spring  Maryland.  AI  decision  aide  learns  by  observing 
only  13  key  fuzzy  parameters.  Each  of  the  attributes  are  linguistically  described  by 
"weak,  moderate,  strong"  etc  and  the  outcome  is  a  probability  of  a  severe  storm  rated 
from  .15  to  95.  This  is  similar  to  discriminate  analysis  except  that  coefficients  are 
heuristically  derived  from  developers  experience.  The  nonlinear  forecast  functions  uses 
technique  from  preference  theory.  He  stresses  that  this  AIDS  the  forecaster  in  doing 
HIS  job. 

Casimir  A.  Kulikowski,  Knowledge  Acquisition  and  Learning  in  EXPERT,  Proceedings  of 
the  International  Machine  Learning  Workshop,  Dept,  of  Computer  Science,  U.  of  Illinois, 
Urbana,  June  22-24  1983. 
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SEEK  provides  a  data  base  of  problem  solving  examples  which  permits:  induce  reasoning 
rules  using  partial  knowledge  of  casual  and  hierarchial  relationships,  suggest  improve¬ 
ments  to  human  rules. 

Alan  Bundy,  Bernard  Silver,  A  Critical  Survey  of  Rule  Learning  Programs,  European 
Conf.  on  AI,  July  1982,  pg  151-157. 

Comparison  of  ELM  and  AMBER  of  Langley,  LEX  of  Mitchell,  Model  Inference  of 
Shapiro,  and  Youngs  extensions  of  Winston  (75).  Authors  find  that  similar  methods  are 
used  to  tackle  similar  problems,  with  every  program  having  a  critic  for  identifying  faults 
and  a  modifier  for  correcting  faults. 

Jaime  G.  Carbonell,  Ryszard  S.  Michalski,  Tom  M.  Mitchell,  Machine  Learning:  A 
Historical  and  Methodological  Analysis,  AI  Magazine,  Fall  1983. 

Good  summary  of  taxonomy  of  machine  learning.  Describes  types  of  learning  and  the  10 
types  of  knowledge  classification  descriptions. 

3ames  A.  Reggia,  Pearl  Y.  Wang,  Dana  S.  Nau,  Minimal  Set  Covers  As  a  Model  for 
Diagnostic  Problem  Solving,  MEDCOMP,  1982,  pg  340-347,  HL. 

This  model  offers  a  framework  for  multiple  simultaneous  disorders  for  a  framework  for  a 
theory  of  diagnostic  inference.  This  is  similar  to  Michalski's  methodology  of  inductive 
learning. 

P.  R.  Innocent,  Towards  Self-Adaptive  Interface  Systems,  Int.  3 ml.  Man-Machine 
Studies,  Academic  Press  Inc.,  16,  1982,  287-299, 

This  paper  follows  a  trend  towards  more  user-oriented  design  approaches  to  interactive 
computer  systems.  The  implicit  goal  in  this  trend  is  the  development  of  more  "natural” 
systems.  The  term  "soft  facade"  is  used  to  describe  a  modifiable  interface  to  a  system. 

Patrick  H.  Winston.  Learning  New  Principles  From  Precedents  and  Exercises,  Artificial 
Intelligence,  19,  3,  1982,  321-350. 


— AIB 


AI  BOOKS 


Elaine  Rich,  Artificial  Intelligence,  McGraw  Hill  $27.,  436,  May  1983,  Univ.  Texas  at 
Austin 

Reader  is  a  programmer,  computer  scientist  or  graduate  student.  This  is  the  only  recent 
book  to  provide  a  balanced  treatment  of  a  majority  of  the  AI  fields. 

Edward  A  Feigenbaum,  Pamela  McCorduck,  The  Fifth  Generation  -  Artificial 
Intelligence  and  Japan's  Computer  Challenge  to  the  World,  March  1983,  275,  Addison 
Wesley  $15.55,  Feigenbaum  is  a  Stanford  expert  in  AI  writing  a  popular  book. 

Japan's  fifth  generation  project  is  described  very  well,  although  critically,  in  50  pages. 
Coverage  of  meager  attempts  by  England  and  France  to  counter  Japan  takes  up  20  pages. 

Trends  and  Applications  :  Automating  Intelligent  Behavior,  Applications  and  Frontiers, 
Proceedings,  May  1983,  IEEE  Cat  83CH1887-9,  NBS 
Excellent  overview  symposium 

<  European  Conference  AI,  1982,  300  aprox,  Yves  Kodratoff,ECAI-82  Conf,  LRI  Batimant 
490,  F91405,  Orsay  CEDEX,  France  $20 

Robert  M.  Glorioso,  C.  Colon  Osorio,  Engineering  Intelligent  Systems: 

Concepts,  Theory,  and  Applications,  Digital  Press,  1980,  472, 


A  comprehensive  overview  showing  how  current  concepts  and  techniques  in  various  fields 
including  electrical  engineering,  mathematics,  and  neuro-physiology,  contribute  to 
designing  intelligent  systems. 

Nils  J.  Nilsson,  Principles  of  Artificial  Intelligence,  Tioga  Publishing,  1980,  476, 

Describes  fundamental  artificial  intelligence  (AI)  ideas  that  underlie  many  of  its 
applications.  The  important  roles  played  in  AI  by  generalized  production  systems  and  the 
predicate  calculus  are  stressed.  The  text  is  designed  for  a  senior  or  first-year  graduate 
course  in  AI. 

Publications  Bibliography  MIT  AI  LAB,  March  1983,  MIT  AI  LAB,  Cambridge,  MA 
02139,  617-233-6218,  AI  Memo  //  191, 


—MAG - MAGAZINES/CONFERENCES 


AI  TESTING 
Artificial  Intelligence 
AI  Magazine  quarterly 

ACM  SIGART  Special  Interest  Group  in  AI  quarterly  magazine  with  excellent  review  of 
current  AI  work  and  publications. 

Journal  of  Cybernetics  now  Cybernetics  and  Systems 

Journal  of  Cybernetics  and  Information  Sciences 

International  Journal  of  Man/Machine  Studies 

Systems,  Man,  and  Cybernetics  Review 

Cybernetics  and  Systems 

Computer  and  Information  Systems  Abstracts 

Machine  Mediated  Learning,  An  International  J.  Voll  //I  Spring  1983 

Electronic  Test 

ATE  Newsletter  NAVY  quarterly  ELEX  08TA  Naval  Electronic  Sys.  Command,  Wash  DC 
20363,  covers  testability,  ATLAS,  Integrated  Diagnostics,  Military  research  groups 

CONFERENCES  (ANNUAL) 

AUTOTESTCON  IEEE 

International  Test  and  Measurement  Conference 
Computer-Aided  Design  IEEE  International  Conference 
CAD-CON  Computer  Aided  Design  Conf.  sponsored  by  magazine 
Total  System  Reliability  Symposium  IEEE 
Reliablity  and  Maintainablity  Symposium  (DEC.  1983) 

International  Symposium  on  Fault  Tolerant  Computing 
National  Aerospace  and  Electronics  Conference 
Computers  in  Aerospace  Conference  IEEE/  AIAA 
Digital  Avionics  Conf  biannual  4th  1981  IEEE  AIAA 
Advanced  Automation,  International  Conference  IEEE 

Computer  Applications  in  Medical  Care  Annual  Symposium  IEEE 
International  Conf  on  Pattern  Recognition 
Proceeding  Pattern  Recognition  and  Image  Processing 
Now  Computer  Vision  and  Pattern  Recog. 

International  Joint  AI  (BIANNUAL)  (..79  81  83) 

International  Conference  on  Cybernetics  10th  =  1983 
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International  Coni  on  System  Sciences,  Hawaii 
Cybernetics  and  Society,  International  Conf.  CIC  Q300  117 


—AIM - AI  MISC 


Special  Issue  on  Knowledge  Engineering,  Computer,  19S3. 

A.  Bolour  et  al,  The  Role  of  Time  in  Information  Processing,  SIGART,  April  1982,  28-48, 
Annotated  bibliography  of  69  papers  that  deal  with  temporal  aspects  of  AI  and 
information  processing.  Instead  of  a  snapshot  description  this  allows  real  world  models 
which  include  history  and  evolution. 

#32  The  Notion  of  Time  in  Medical  Records  Methods  of  Information  in  Med  4/78 
#51  An  Interval-based  representation  of  Temporal  Knowledge  IJAI  August  81 
#63  TERM:  An  approach  to  include  the  time  dimension  in  the  entity-relationship  model 
Second  Int.  Conf.  on  the  Entity  Relationship  Oct  1981 

James  F.  Allen,  Maintaining  Knowledge  about  Temporal  Intervals,  Communications  of 
the  ACM,  November  1983,  832-843,  Univ.  of  Rochester 

Interval-based  temporal  logic  is  developed  and  examples  are  given  for  the  following 
database  types:  historical  data,  modeling  processes  and  process  interaction,  and  inter¬ 
active  system  where  present  moment  is  continually  updated. 

The  Current  State  of  Ah  One  Man's  Opinion,  Roger  C.  Schank,  Yale  University,  AI 
Magazine.  Winter/Spring  1983,  3-8, 

David  Waltz  Chairman,  Artificial  Intelligence^  Assessment  of  the  State-of-the-Art 
and  Recommendation  for  Future  Directions,  AI  Magazine,  Fall  1983, 

In  depth  analysis  of  natural  language  and  expert  systems  with  recommendations  for  next 
5-10  years. 

A.  A.  Araya,  Learning  Modalities  in  Artificial  Intelligence  Systems:  A  Framework  and 
Review,  Proceedings  of  the  2nd  International  Conference  on  Computer  Science,  August 
1982.  54-84 

Intelligent  systems  should  possess  two  fundamental  capabilities:  problem  solving  and 
learning.  Problem  solving  capabilities  allow  an  intelligent  system  to  cope  with  problems 
in  a  given  domain.  Learning  capabilities  make  it  possible  for  an  intelligent  system  to 
improve  the  solution  to  problems  within  its  current  reach,  or  to  cope  with  new  problems. 
This  paper  examines  research  in  Artificial  Intelligence  from  the  perspective  of  'learning' 
with  the  purpose  of:  1)  developing  and  understanding  of  the  problem  of  learning  from  the 
AI  point  of  view,  and  2)  characterizing  the  current  state  of  the  art  on  learning  in  Al.  (35 
refs.) 

R.  B.  Banerji,  Theory  of  Problem  Solving:  A  Branch  of  Artificial  Intelligence,  Proc.  IEEE, 
December  1982,  1428-1448,  GPS 

Judea  Pearl,  Antonio  Leal,  Joseph  Saleh.  GODDESS:  A  Goal-Directed  Decision 
Structuring  System.  IEEE  Transactions  on  Pattern  Analysis  and  Machine  Intelligence. 
PAMI-4,  3,  May  1982,  250-262 

D.  G.  Shapiro,  Sniffer:  A  System  that  Understands  Bugs  June  1981,  AD-A102158,  MIT 
AIM  638, 

This  Masters  thesis  develops  expert  system  for  software  program  debugging  by  having  an 
assortment  of  experts  which  can  be  applied  to  suspect  code.  This  might  also  be  useful 
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for  V/V  of  test  program  software  to  see  that  all  of  the  constraints  have  been  met: 
accuracy,  timing,  error  callout,  comments  etc. 

D.  A.  McAllester,  Reasoning  Utility  Package  User's  Manual,  April  1982,  AD-A1 14756, 
MIT  AIM  667, 

Contains  a  Truth  Maintenance  Package  (TMS)  which  allows  separation  of  assumptions 
from  solid  facts  and  keeps  track  of  the  effect  of  changes  to  an  expert  system. 

B.  Chandrasekaran,  Toward  a  Taxonomy  of  Problem  Solving  Types,  AI  Magazine,  9-17, 
Winter/  Spring  1983, 

M.  S.  Y.  Wang,  Keh-Chiang  Yu,  A  Hierarchical  View  of  Decision  Support  System 
Software,  Proceeding  of  16th  conf  on  System  Sciences  1983,  482-489,  Univ  of 
Texas/ Austin 

Youji  Fukada,  Primary  Algorithm  for  the  Understanding  of  Logic  Circuit  Diagrams, 
Pattern  Recognition  Conf  1982  IEEE,  706-709,  Mitsubishi  Japan. 

Inputs  schematic  diagrams  into  computer  from  drawings. 

G.  Neugebauer,  DOCUMENT  ANALYSIS  The  TEXT  SCANNER:  A  Model  for  Analyzing 
and  Classifying  Document  Elements,  Pattern  Recognition  Conf  1982  IEEE,  486-488, 
Siemens,  Pattern  Recognition 

John  W.  Verity,  LISP  Markets  Grow,  Datamation,  Oct  1983,  92-100 

While  Symbolics,  Xerox  and  LISP  Machine  Inc.  control  the  $50  Million  market  for 
personal  LISP  machines,  a  dozen  or  so  startups  in  the  artificial  intelligence  field  are 
fueling  interest  in  new  LISP  tools.  Names  many  new  «-nall  AI  companies. 

Randall  Davis,  Interactive  Transfer  of  Expertise:  Acquisition  of  New  Inference  Rules, 
Artificial  Intelligence,  12,  1979,  1210157 

TEIRESIAS  is  a  program  designed  to  provide  assistance  on  the  task  of  building 
knowledge-based  systems.  It  facilitates  the  interactive  transfer  of  knowledge  from  a 
human  expert  to  the  system,  in  a  high  level  dialog  conducted  in  a  restricted  subset  of 
natural  language.  The  concept  of  meta-level  knowledge  is  described  and  illustrations  are 
given  of  its  utility  in  knowledge  acquisition  and  its  contribution  to  the  more  general 
issues  of  creating  an  intelligent  program. 

James  B.  Schultz,  Weapons  That  Think,  Defense  Electronics,  January  1983,  74-80 
Pentagon  analysts  look  to  AI  to  solve  problems  in:  1)  C3I  Fusion  (information  reduction), 
2)  Missile  Guidance,  3)  Robotic  Tanks,  4)  Intelligence  (translate  language,  read  in 
interpretations  based  on  dialects,  local  phraseology,  and  inflections)  5)  Electronic 
Warfare. 

Michael  Georgeff,  Strategic  Search,  Australia  Computer  Science  Communications,  2,  1, 
Jan  1980,  25-38 

Strategic  search  out  performs  heuristic  searching. 

Edward  C.  Taylor,  Artificial  Intelligence  in  the  Air-Land  Battle.  Astronautics  <5c 
Aeronautics,  July/ August  1983,  55-59,  AIAA,  TRW  Defense  Systems  Group 
The  key  limit  in  a  real  time  intelligence  system  is  not  the  volume  of  data  the  analysts 
can  get  to,  but  rather  the  mental  limits  of  the  analysts.  This  paper  briefly  reviews 
several  knowledge  engineering  and  AI  systems,  noting  the  similarities  between 
capabilities  of  existing  systems  and  the  needs  of  military  situation  assessment.  The 
author  concludes  that  development  of  an  operational  military  expert  system  is  a  tough 
challenge,  but  is  technologically  and  culturally  feasible. 
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BIT 


- BIT/ATE  - 


A  J.  Kunert,  ATE  Applications  of  Artificial  Intelligence,  1982  IEEE  AUTOTESTCON,  84- 
87,  Naval  Air  Eng.  Center, 

The  author  does  not  see  near  term  uses  of  AI  for  ATE  due  to  memory  and  speed 
limitations.  The  Navy  would  like  friendlier  ATE  that  would  have  a  dialogue  with  an 
operator  while  testing.  An  Expert  Diagnostician  would  have  knowledge  of  previous  faults 
and  causes  which  it  could  apply  to  future  diagnosis.  Self  improving  diagnostics  are 
alluded  to  in  addition  to  BIT  and  monitor  IC,  as  well  as  CAD  systems  which  include  a 
testability  review  which  are  aware  of  the  capabilities  of  the  target  ATE. 

Institute  for  Defense  Analysis,  Built-in-Test  Requirements  Workshop  IDA  1981  P-1600 
219  Landmark  paper,  widely  referred  to. 

Betty  Sponaugle,  ATE  NEWSLETTER,  ELEX  08TA  Naval  Electronic  Systems  Command 
DC  20363. 

Quarterly  newsletter  of  Automatic  Test  Equipment  news  in  DoD.  Covers  testability, 
ATLAS,  MATE,  VHSIC  Self-test,  Army  Diagnostic  Technology. 

D.  E.  Tuttle,  R.  Loveless,  Built-In-Test  and  External  Tester  Reliability  Characteristics, 
March  1980,  AD-A083-488,  RADC  TR-80-32,  Lockheed 

3.  G.  Malcolm,  BIT  False  Alarms:  An  Important  Factor  in  Operational  Readiness, 
Proceed.  Reliability  and  Maintainability  Symposium.  IEEE,  1982,  206-212,  Hughes, 
Excellent  paper  on  BIT  false  alarms,  including  AI  solution  ideas. 

Robert  T.  Chien,  Roderick  3.  Fletcher,  William  P.  C.  Ho,  Larry  3.  Peterson,  On  the  Use 
of  Procedural  Models  for  Generation  of  Test  Programs,  IEEE  Transactions  on  Circuits 
and  Systems,  July  1979,  IEEE,  CAS-26,  7,  555-557. 

An  approach  based  on  artificial  intelligence  concepts  is  developed  for  automatic 
generation  of  test  programs  for  analog  circuits.  The  programs  will,  with  the  help  of 
automatic  test  equipment  (ATE),  make  appropriate  measurements  and  deduce  the 
location  of  potential  faults  in  analog  circuit  boards. 

J.  G.  Malcolm,  Practical  Application  of  Bayes'  Formula,  Reliability  and  Maintainability 
Annual  Proceedings,  1983,  180-186,  Hughes 

The  author  'proves'  that  false  alarms  cannot  be  eliminated  for  the  single  measurement 
case. 


J.  G.  Malcolm,  G.  L.  Foreman,  The  Need:  Improved  Diagnostics— Rather  than  Improved 
Reliability,  Proceedings  Annual  Reliability  and  Maintainability  Symposium  1984,  Jan 
1984,  315-322,  Hughes. 

This  paper  describes  an  extension  of  Malcom's  Bayesian  analysis  in  which  a  Bayesian 
processor  of  fault  information  was  simulated  and  found  to  greatly  reduce  the  false  alarm 
rate.  This  is  an  extension  of  the  "n  of  m"  filtering  of  alarms,  which  the  authors  claim 
does  not  have  enough  statistical  independence.  They  show  that  it  is  wrong  to  heavily 
filter  a  BIT  circuit,  and  that  it  is  far  better  to  apply  statistical  processing  to  the  alarm 
output. 

They  define  SMART  BIT  as  any  BIT  that  approaches  the  ideal  of: 

1)  100%  diagnostic  capability 

2)  Unambiguous  fault  isolation  to  the  module  level  3)  Non-volatiie  memory 

4)  No  false  alarms 

5)  The  ability  to  detect/isolate  intermittent  faults 
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Edward  3.  McCluskey,  S.  Bozorgui-Nesbat,  Design  for  Autonomous  Test,  IEEE  Transac¬ 
tions  on  Circuits  and  Systems.  November  1981,  1070-1078,  Stanford  Center  for  Reliable 
computing. 

Technique  for  modifying  networks  so  that  they  are  capable  of  self  test  is  presented. 
Innovation  in  partioning  into  subnetworks  with  sufficiently  few  inputs  that  exhaustive 
testing  of  subnetwork  is  possible. 

Test  Technology  Information  Center  Fleet  Analysis  Center,  Naval,  Weapons  Station, 
Corona  Calif.,  Built-In  Test  (BIT)  Equipment  and  Methods,  Report  824-1619,  Oct  15  1981, 
150  aprox,  Literature  search  for  "BIT"  etc 

George  Neuman,  Navy  Testing  Technology  Strategy  Team,  Built-In-Test  Effectiveness 
Study,  USN  1124/0679/BD14-2,  June  1983,  70  aprox,  Testing  Technology  Fleet  Analysis 
Center  NWS  Corona  Beach,  Giordano  Associates, 

Test  Technology  Information  Center,  Built-In  Test/  Testability  Improvement  Program, 
TTIC  Fleet  Analysis  Center  Code  8242  Corona  CA  91720,  TM-824-1658,  Aug  83  Draft 
Up  to  date  100+  page  analysis  of  BIT  and  T  for  all  of  DoD  including  guides,  MATE, 
directives,  standards,  and  their  interrelationships. 

Thomas  G.  Freund,  Applying  Knowledge  Engineering  to  TPS  Development, 
AUTOTESTCON-  IEEE.  October  1983,  Harris 

He  states  that  most  development  of  software  production  tools  has  concentrated  on 
production  phase  which  contributes  between  1096-2096  of  the  toted  TPS  cost.  An 
additional  50%  of  the  costs  come  from  the  design  and  requirements  phases  of  TPS.  They 
are  adapting  the  Programmer's  Apprentice  to  make  the  TPS  Assistant.  This  is  a  part  of 
DATE(  Development  Assistant  for  TPS  Engineering). 

L.  F.  Moebus,  Reducing  Military  Digital  Testing  Life  Cycle  Costs,  “AUTOTESTCON  - 
IEEE,  October  1983,  General  Dynamics 

Looking  at  the  digital  test  portion  of  military  ATE  system  they  find  the  50%  of  the  LCC 
is  for  operators  and  maintenance  personnel,  and  only  9%  is  due  to  test  program  set 
design.  "Interactive  processor  testing  appears  to  be  the  future  backbone  of  military 
digital  testing."  Much  more-off  loading  of  the  test  to  the  internal  BIT. 

Charles  A  Locurto,  Impact  of  BIT  on  Avionics  Maintainability,  1983  Proceedings  Annual 
Reliability  and  Maintainability  Symp,  333-338,  From  his  PHD  dissertation  at  South- 
Eastern  Univ  Nov  1982 

A.C.  MacMurray,  V.  D.  Leonard,  On-Line  Monitoring  Guide  for  Technical  Personnel,  TD 
579,  March  1983,  Naval  Ocean  System  Center-  San  Diego 

A.C.  MacMurray,  V.  D.  Leonard,  Management  Personnel  Performance  Monitoring  Guide, 
TD  575  March  1983,  Naval  Ocean  System  Center-  San  Diego,  has  program  milestone 
charts  for  BIT. 

R.  K.  Nair,  Diagnosis,  Self-Diagnosis  and  Roving  Diagnosis  in  Distributed  Systems,  AD- 
A069770,  Sept  1978,  Univ  of  Illinois-Urbana 

Thesis  describes  the  type  of  system  for  which  roving  diagnosis  is  possible.  This  is  self 
testing  without  external  control.  He  also  discusses  how  this  minimizes  testing  time.  This 
is  an  important  concept  for  System  Integrated  Test  or  SMART  BIT. 

BIT/ETE  Figure  of  Merit  and  Demonstration  Techniques,  1979,  RADC-  Hughes 

BIT  analysis  is  very  costly:  $1,000,000.  Has  most  extensive  set  of  BIT  definitions  (18). 

Demonstration  includes  transient  insertion. 
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J.  McDermott,  Stand-Alone  Function-Generator  ICS  Extend  Design  Features,  EDN,  28 
#12,  61-67,  1983 

In  the  application  of  design  for  testability,  describes  the  use  of  a  function  generator  IC 
able  to  output  sine,  triangle,  ramp,  and  tone  burst,  either  at  discrete  frequencies  or 
swept.  Useful  for  Smart  BIT. 

R.  Owen  Holbrook,  BIT  Detectability/Reliability  in  Expendable  Weapons,  Proceedings 
Annual  Reliability  and  Maintainability  Symposium,  1984 

This  and  his  1983  paper  are  among  the  few  which  have  equations  for  BIT  on  system 
performance,  costs  etc.  He  gives  an  example  showing  an  equivalent  increase  of  MTBF  by 
5X  by  having  a  BIT  detectability  of  75%  at  the  end  of  4  years  for  a  weapon  normally  in 
storage.  The  effect  on  availability  for  2000  weapons  is  a  difference  of  1400  for  no-BIT 
and  1600  with  BIT.  Thus  you  get  200  more  weapons  for  the  cost  of  the  BIT.  He  shows  that 
there  is,  however,  a  higher  maintenance  cost  of  14%  for  this  example. 

Bruce  L.  Havlicsek,  Gerald  G.  L.  Meyer.  The  Development  and  Application  of  System- 
Level  Fault  Models,  Proceedings  of  Automatic  Test  Program  Generation  Workshop.  1983. 
IEEE  Computer  Society,  is  with  the  Integrated  Logistics  Support  Division. 

Meyer  is  with  Johns  Hopkins  University. 

This  paper  presents  a  new  approach  to  determine  conditions  that  ensure  diagnosability 
properties  in  complex  systems.  The  approach  uses  a  new  system-level  fault  model 
structure  having  both  internal  and  observable  test  outcomes  and  allowing  multiple  test 
outcomes  to  be  associated  with  each  fault  situation. 

This  paper  also  outlines  an  overall  research  and  development  program  for  the  use  of 
system-level  fault  models  for  real  engineering  applications.  The  program  organizes  the 
individual  aspects  of  fault  model  application  into  a  tractable  and  integrated  proposal. 

Tim  M.  Strike,  Gary  E.  Sanders,  The  use  of  EEPROMs  in  Embedded  Computers  for 
Avionics,  Aerospace  Electronics  Proceedings,  1983,  611,  IEEE 

R.  G.  Bennets,  Introduction  to  Digital  Board  Testing,  1982,  Crane,  Russack  &  Company, 
Inc. 

V.  D.  Agrawal,  S.  K.  Jain,  D.  M.  Singer,  Automation  in  Design  for  Testability,  Custom 
Integrated  Circuits  Conference.  1984.  AT&T  Bell  Laboratories 

Through  a  well-defined  set  of  design  rules,  the  scan  approach  allows  complete  automa¬ 
tion  in  chip  testability.  This  paper  describes  a  design  automation  system,  TITUS,  which 
automatically  checks  the  circuit  for  design  rules,  implements  the  testability  hardware, 
and  generates  tests. 

Smith  Freeman,  Statistical  Inference  Algorithims  for  Automating  Testing  of  Analog 
Circuits,  Large  Scale  Systems  Symposium  IEEE  1982,  32-39,  RCA 

1983  IEEE  ATPG  Workshop  Proceedings,  IEEE  Catalog  #830H  1878-8,  133  pages,  20 
articles. 

Curriculum  for  Test  Technology,  IEEE  Catalog  #  83CH1978-6,  contains  31  pages  on  DFT 
and  Test  Techniques  including  BIT  state-of-the-art  report. 
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G.  G.  Hendrix,  KLAUS:  A  system  for  Managing  Information  and  Computational 
Resources,  SRI  International,  TN  230  SRI  Project  1894,  34 


Overview  of  goals  and  philosophy  of  SRI  work  for  support  systems  than  can  be  tutored  in 
English  about  new  subject  areas.  Useful  in  filing  and  retrieving  information  and  allowing 
access  to  other  tools  (DBMS,  planner,  schedulers,  report  gen.  etc)  Excellent  overview. 

R.  W.  Allen,  C.  D.  Chen,  M.  M.  Ervin-Willis,  K.  R.  Rahlfs,  R.  E.  Tulloss,  S.  L.  Wu, 
DORA:  A  System  of  CAD  Post-Processors  Providing  Test  Programs  and  Automatic 
Diagnostics  Data  for  Digital  Device  and  Board  Manufacture,  Western  Electric  Company, 
1981  Test  Conference.  1981,  555-560 

Jeffry  Beeler,  IBM,  Stanford  Join  Forces  on  AI  Project,  Researchers  Working  on  AI  for 
Diagnosing  Failures  in  Intrasystem  Hardware,  November  16,  1981,  COMPUTERWORLD, 
Describes  the  use  of  AI  to  diagnose  intrasystem  hard-ware  failures  in  a  project  jointly 
sponsored  by  DEC  and  MIT 

T.  R.  Addis,  Knowledge  Refining  for  a  Diagnostic  Aid  (An  Example  of  Applied 
Epistemics),  International  Journal  Man-Machine  Studies,  v!7,  August  1982,  151-164 
Epistemics  is  the  science  of  communicating  understanding  via  stored  knowledge. 
Knowledge  refining  is  the  elimination  of  redundant  descriptions  and  the  exposure  of 
contradictory  or  missing  information,  so  that  the  essence  of  a  structured  activity  can  be 
captured  for  computer  manipulation.  Tries  to  describe  activity  like  DIAGNOSTICS  in 
the  form  of  a  hierarchy  of  actions.  This  hierarchy  can  be  constructed  by  the  recursive 
application  of  a  pattern  recognition  technique(RAFFLES).  Knowledge  refining  provides 
AUTOMATIC  KNOWLEDGE  MAINTENANCE  for  an  expert  system.  See  also  Addis  work 
in  Maintenance  Expert  section. 

Ramesh  S.  Patil  etc,  Causal  Understanding  of  Patient  Illness  in  Medical  Diagnosis,  IJAI 
81,  893-899,  MIT 

Allows  multivariate  relation  between  causes  and  effects  which  varies  with  context.  If  at 
some  level  of  detail  two  distant  phenomena  interact,  they  aggregate  the  description  so 
that  the  two  phenomena  are  adjacent.  Aggregation  can  be  focal  (the  nodes  of 
importance)  or  causal. 

Michael  R.  Genesereth,  Diagnosis  Using  Hierarchical  Design  Models,  (DART)  278-283, 
AAAI-82,  Stanford  University 

Abstract;  This  paper  presents  a  new  algorithm  for  the  diagnosis  of  computer  hardware 
faults.  The  algorithm  uses  a  general  inference  procedure  to  compute  suspect  compon¬ 
ents  and  generate  discriminatory  tests  from  information  about  the  design  of  the  device 
being  diagnosed.  In  the  current  implementation  this  procedure  is  linear-input  resolution, 
guided  by  explicit  meta-level  control  rules.  The  algorithm  exploits  the  hierarchy 
inherent  in  most  computer  system  designs  to  diagnose  systems  a  level  at  a  time.  In  this 
way  the  number  of  parts  under  consideration  at  any  one  time  is  kept  small,  and  the  cost 
of  test  generation  remains  manageable.  Uses  common  LISP  with  data  base  and  inference 
system  (MRS).  It  runs  on  VAX  11-780  in  minutes  regardless  of  the  size  of  the  model! 

Hal  Shubin,  John  Wade  Ulrich,  IDT:  An  Intelligent  Diagnostic  Tool,  AAAI-82,  290-295 
Abstract:  IDT  is  an  intelligent  hardware  diagnostic  tool  that  has  been  successfully  used 
to  identify  faults  in  PDP  11/03  computers.  It  selects  and  executes  tests,  and  interprets 
the  results.  IDT  is  able  to  modify  its  test  selection  strategy  on  the  basis  of  results  of 
previous  tests  as  well  an  opinions  offered  to  it  by  the  user.  Symbolic  formulas  are  used 
to  represent  the  relationship  between  the  test  results  and  the  broken  equipment.  If  we 
assume  that  there  is  only  one  broken  component,  then  set  operations  can  be  shown  to  be 
sufficiently  general  for  combining  the  results  of  multiple  tests. 


Thomas  Whalen,  Brain  Schott,  Fred  Ganoe,  Fault  Diagnosis  in  a  Fuzzy  Network  IEEE 
1982  Cybernetics  and  Society,  28-30  October  1982,  35-39 

This  paper  presents  a  method  for  detecting  and  diagnosing  departures  from  optimal 
output  in  a  fuzzy  network,  by  means  of  a  fuzzy  production  system  (a  computer-assisted 
algorithm  for  making  deductions  from  fuzzy  premises).  A  human  decision  maker 
interacts  with  the  production  system  through  a  quasi-natural  language  system  which  has 
been  implemented  in  APL. 

B.  Chandrasekaran,  F.  Gomez,  S.  Mittal,  3.  Smith,  M.D..  An  Approach  to  Medical 
Diagnosis  Based  on  Conceptual  Structures,  6th  I3CAI,  1979,  Tokyo  Japan.  134-142 
Some  principles  governing  the  design  of  conceptual  structures  are  presented.  We  apply 
the  ideas  to  the  design  of  MDX. 

Joseph  G.  Wohl,  Cognitive  Capability  VS  System  Complexity  in  Electronic  Maintenance, 
Cybernetics  and  Society  Proceedings.  1982,  IEEE,  370-372,  ALPHATECH 
Condensation  of  another  paper  by  Wohl 

P.  P.  Bennetts,  Failure  Diagnosis  and  Decision  Making  in  Industrial  Processes:  A  Fuzzy 
Set  Application,  IEEE,  1087-1093,  General  Electric 

National  Security  Industry  Association:  Automatic  Test  Committee  and  Logistics  Man¬ 
agement  Committee,  Conference  on  Integrated  Diagnostics.  Integrated  Diagnostic 
Conference,  Feb  1983,  NSIA,  Tampa  Florida,  Most  complete  information  on  Integrated 
Diagnostics. 

Charles  Curtis  National  Security  Industrial  Association 
1012-15th  St  NW,  Suite  901,  Wash  DC  20005  202-393-3620 

Roger  T.  Hartley,  CRIB:  Computer  Fault-finding  Through  Knowledge  Engineering 
Computer,  March  1984,  Kansas  State  University 

Good  example  of  maintenance  expert  system  with  ability  to  learn  diagnostics. 

Consideration  for  the  Systematic  Design  of  Machine  Self-Diagnosis,  Brian  A.  Kelley, 
Richard  D.  Rosen,  May  1983,  International  Appliance  Technical  Conference  Purdue 
Univ.,  Battelle-Columbus 

Their  work  is  still  at  the  design  stage.  They  are  considering  sensor  selection  and 
requirements.  They  have  incorporated  some  of  the  techniques  of  Pau,  who  works  at 
Batteile  in  Switzerland.  Their  interest  is  in  mechanical  systems  diagnosis,  especially  jet 
engines, 

Mike  Partridge,  Integrated  Testing  and  Maintenance  Technologies,  December  1983, 
AFWAL  TR-83-1183 

Contract  with  Boeing  from  Air  Force  Wright  Aeronautical  Lab  to  define  the  require¬ 
ments  for  onboard  test  system  for  the  avionic  suite  for  the  tactical  fighters  in  the 
1990's.  Identified  improvements  over  current  systems  include  better  filtering  of  inter¬ 
mittent  failure  reports,  intermittent  fault  isolation  through  the  use  of  recorded  data, 
more  extensive  use  of  system-level  tests  of  mission  operational  data  and  a  man-machine 
interface  providing  more  information  to  the  maintenance  technician.  In  addition,  a 
design  concept  for  a  fault  classification  expert  system  was  developed. 

L.M.  Carrier  et.  al.  Operational  Flight  Software  Techniques  for  On-Aircraft  Testing  and 
Maintenance,  Fifth  Digital  Avionics  Conference.  1983.  Discusses  onboard  fault  detection 


R.  K.  Kandt,  Ronald  Newlon, Self-Improving  Diagnostics  for  Automatic  Test  Equipment, 
3 an  1981,  7,  ATE  Seminar/Exhibit  Proceedings,  Electronics  Test  Magazine,  Hughes 
This  is  the  result  of  the  MATE  and  RADC  work  with  KBS.  This  had  43  rules  in  Emycin 
which  the  user  could  add  to  incrementally  to  improve  the  behavior  of  the  diagnostics. 
The  system  would  allow  user  to  enter  faults,  probability,  and  procedures  to  correct  the 
fault,  along  with  a  certainty  factor.  A  much  more  descriptive  phrase  would  be  user- 
improvable  diagnostics'. 

Charles  Paulson,  Classes  of  Diagnostic  Tests,  20th  Design  Automation  Conference,  1983, 
316-322,  Stanford  Computer  Science  Dept. 

SUBTLE  is  a  predicate  calculus  language  for  combinational  digital  systems.  Work  is  in 
conjunction  with  IBM  and  Fairchild  and  early  DART  work. 

B.  L.  Havlicsek,  G.  -G.  Meyer,  A  New  System-Level  Approach  to  Diagnosability, 
JHU/EECS-82/6,  1982,  Johns  Hopkins  Univ/  funded  by  ONR  and  Westinghouse 
Does  not  require  fault/test  relationship.  Permits  multiple  test  outcomes  with  each  fault. 
Internal  (unobservable)  test  outcomes  can  be  used.  When  the  system  has  a  Morphic 
property  it  becomes  orders  of  magnitude  easier  to  describe.  This  is  similar  to  graphical 
fault  model  description.  Report  is  highly  theoretical. 

John  Reviore,  Reliability  and  Maintainability  Volume  6.1 -Diagnostics,  Institute  For 
Defense  Analysis,  Volume  in,  1-64,  August  1983 

Volume  III  is  a  summary  document  of  some  42  other  volumes  of  a  DoD  R&M 
improvement  study.  There  are  significant  differences  in  definition  of  False  Alarms(pg  4). 
FA  impact(pgl3)  includes  operator  ignoring  BIT  indications,  excessive  LRU  swapping 
which  leads  to  maintenance  induced  failures.  It  reviews  many  of  the  recent  reports  in 
this  field. 

An  air  crew  may  need  fault  detection  in  terms  of  seconds.  Fault  isolation  of  environ¬ 
mentally  mission  induced  problems  require  the  system  to  remember  why  the  failure 
occurred  (pg36).  Maintenance  action  times  of  4  hours  typical  are  accepted  normally,  but 
there  would  be  considerable  delay  in  maintenance  in  a  high  sortie  generation  schedule. 
Minimizing  Bench  Check  Serviceable  is  important  for  three  level  maintenance  concept, 
but  is  an  absolute  necessity  for  two-level  concept  (pg44). 

Diagnostics  section  includes  the  following  conclusions.  Specify  R&M  in  terms  of 
exceeding  a  threshold  of  $XX.XX  acquisition  cost  per  removal-free  operating  hour 
(difficult  to  communicate  to  designer  pg  49).  Design  needs  to  have  a  flexible  diagnostic 
system  so  that  changes  can  be  incorporated  readily  in  diagnostic  algorithms,  screens  and 
tolerances  with  minimal  hardware  impact.  Techniques  are  needed  to  enable  more 
concurrent  hardware  and  software  development  and  earlier  integration  of  the  two.  CAE 
techniques  for  enhancing  design  for  testability  in  support  of  proposed  MIL-STD-XXX. 
Diagnostic  testing  in  the  early  operations  environment  will  reveal  BIT  problems. 
Recognize  that  maturation  phase  is  a  legitimate  design  activity  and  plan  for  it. 
Operational  Test  and  Field  Maturation  should  include  a  special  diagnostic  data  collection 
and  analysis  system  to  capture  information  on  occurrences  and  causes  in  enough  detail  to 
provide  a  credible  data  base  for  developing  and  implementing  engineering  solutions. 

Conference  on  Integrated  Diagnostics  National  Security  Industry  Association:  Automatic 
Test  Committee  and  Logistics  Management  Committee,  February  1983,  230,  Latest 
information  on  BIT,  etc.  Partial  table  of  contents: 

Overview  of  Maintenance  Aiding,  XYZYX  Corp 

Technology  and  Technology  Gaps  between  present  and  future  performance  aids 
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Artificial  Intelligence  Applications  to  Maint.  RADC  Coppola" 

New  look  at  BIT  Hughes  Malcolm 

Diagnostic  Technology  NSWC  Keiner 

Consolidated  Support  System  NASC  Reagan 

Status  of  maintenance  training  simulation  research  in  the  USAF  Pohiman 

ATE  State  of  the  art  in  training  technology  TTA  Halley 

Futuristic  Maintenance  Trainers  NTEC  Rizzo 

Integration  of  Automated  test,  maintenance  aiding 


and  maintenancetraining  Biotechnology  Post 

Ben  Moszkowski,  Reasoning  about  Digital  Circuits,  STAN  -CS-83-970,  July  1983,  146, 
Stanford  University 

Enhancement  of  predicate  logic  with  interval  temporal  logic  (ITL)  to  accommodate  time- 
dependent  operators.  Is  able  to  describe  latches,  counters,  RAM,  and  bit  slice. 

John  A.  Swets,  Ronald  M.  Pickett,  Evaluation  of  Diagnostic  Systems:  Methods  from 
Signal  Detection  Theory,  Academic  Press,  1982,  253,  Bolt,  Beranek,  and  Newman 
Using  Relative  operating  Characteristics  (ROC)  for  general  diagnosis.  ROC  techniques 
have  been  used  in  signal  detection,  human  perception,  military  monitoring,  etc.  This  is 
possibly  useful  only  to  someone  well  into  diagnosis  problems. 

Alan  Willsky,  A  Survey  of  Design  Methods  for  Failure  Detection  in  Dynamic  Systems, 
Automatica,  12,  1976,  601-611,  Pergamon  Press 
Diagnosis,  detect  abrupt  changes  in  linear  systems 

Edward  Y.  Chow,  Alan  Willsky,  Sequential  Decision  Rules  for  Failure  Detection,  Joint 
Automatic  Control  Conf,  1981,  11, 

Chow  works  at  Schlumberger-Doll  Willsky  at  MIT  Infor  and  Dec.,  detection  by  Bayes 
sequential  decision  problem.  Probably  Chow  PHD  thesis  of  1980 

B.  K.  Walker,  Performance  Evaluation  of  Systems  that  Include  Fault  Diagnosis,  Joint 
Automatic  Control  Conference,  1981,  theoretical 

R.  N.  Clark,  D.  C.  Fosth,  V.  M.  Walton,  Detecting  Instrument  Malfunctions  in  Control 
Systems,  IEEE  Transactions  on  Aerospace  and  Electronic  Systems,  July  1975,  465-473, 
diagnosis 

Kazuo  Nakajima,  A  New  Approach  to  System  Diagnosis,  19th  Allerton  Conf.  on 
Communications,  Control  and  Computing  IEEE,  1981 

Ruey-wen  Liu,  A  Theory  of  Analog  Fault  Diagnosis  and  its  Implementation.  Proceedings 
of  Automatic  Test  Program  Generation  Workshop,  1983,  IEEE  Computer  Society,  83CH 
1878-8,  University  of  Notre  Dame 

This  paper  discusses  two  major  issues  of  analog  fault  diagnosis:  tolerance,  and  modeling 
and  simulation  of  faulty  components.  It  further  discusses  three  measures  of  the 
effectiveness  of  a  testing  program:  test  points,  post-fault  computation,  and  robustness. 


T - TESTABILITY 


Thomas  W.  Williams,  Kenneth  P.  Parker,  Design  for  Testability  -  A  Survey,  IEEE 
Transactions  on  Computers,  January  1982,  2-15,  114  references,  Excellent  Review 


E.  Shemeta,  R.  Spillman,  Computer  Aided  Testability  and  Design  Analysis,  Boeing 
Aerospace,  RADC-TR-83-257,  140  aprox 

R.  Spillman,  Advanced  Application  of  the  Printed  Circuit  Board  Testability  Design  and 
Rating  System,  Boeing  Aerospace,  RADC-TR-83-291,  100+ 

George  Neumann,  Giordano  Associates  TESTING  TECHNOLOGY  (Improving  Weapon 
System  Reliability  and  Maintainability  Study  Program),  Institute  for  Defense  Analyses 
1801N  Beauregard  St.  Alexandrai  VA  April  1  1983 

Tries  to  establish  requirements  for  strong  test  technology  program  for  DOD.  Commer¬ 
cial  IRJcD  efforts  have  been  directed  to  improvements  of  commercial  ATE,  not 
embedded  test.  The  services  have  installed  over  1000  different  ATE. 

BIT/TEST  Improvement  Program  of  Joint  Services,  pg3-2 

Details  of  many  aspects  of  program  are  given  in  this  paper.  They  cover: 

Weapon  System  Testability  Design  Techniques 
CAD/  CAT  Design  Tools 
Logistic  Support  Activity  (LSA)  Process 
T  Prediction  and  Demonstration 
BIT/MTE/ATE/FOMS 
Fault-Tolerant  Design 
On-Line  Testing 

Performance  Monitoring  Built-In-Test  Maint  Aids 
Non-Electronic  Monitoring 
Off-Line  Testing 

ATE  Applications  SW  ATPG  Metrology/Calibration 
Test  Techniques 

Diagnostics/Prognostics  Advanced  Device  Testing 
Non-Conventional  Testing  System-Level  Testing 
Test  and  Evaluation 

Test  Bed  Demo  Experimental  Demonstrations 
Summary  of  guides  pg  4-28  Sponsors  pg  4-31  Performing  Activities  pg4-32 
INTERFACES  TO  OTHER  TECHNOLOGIES  from  testing  technology  PG  4-49 
CAD/CAM  VHSIC  RELIABILITY  FIBER  OPTICS 
DIAGNOSTICS  MECH.  SYSTEMS  CONDITION  MONITOR 
OPERATIONAL  SW  AI  MANPOWER  AND  TRAINING 
PACKAGING  NONDESTRUCTIVE  TESTING 
TRADE-OFFS  DRIVEN  BY  COST 

MTBF  MTTR  MLOGISTIC  DELAY  TIME  PERSONNEL 
PERSONNEL  SKILLS  SPARES/FACILITIES/SPACE 
BIT  VS  Off-Line  Testing  Organizational,  Interm  and  Depot  alloc 
Acquisition  Cost/Performance  Capability  Mobility  Envr. 

LIFE  CYCLE  COST  REDUCTION  of  10%  to  20%  with  good  T  design  pg  5-12 

Anthony  Coppola  Artificial  Intelligence  Applications  to  Maintenance  Report  for 
OSD/IDA  RAM  Study,  April  15  1983  review  copy, 

23  with  30  pg  of  position  papers  —  see  section  8.4.2  of  this  report. 

Anthony  Coppola,  Lorraine  M.  Gozzo,  Artificial  Intelligence  Applications  to  Testability, 
AUTOTESTCON  1982.  IEEE,  138-142,  RADC. 

A.  C.  MacMurray,  U.S.  NAVY  Test  Technology  RDT&E  Plan  (Rev  VO,  NOSC  619-225- 
6173,  June  15  1983,  60+ 


Details  reseach  plan  for  Navy  testing  technology.  Includes  names  of  people  and  programs 
funded  or  planned.  Some  examples: 

ATE  Self  Test  Software 

Complexity,  Testability,  and  Fault  Analysis  of  Electronic  Systems 
Testability  of  Structures 
Testability  Figures  of  Merit 

Air  Launched  Missile  Testability  (unfunded  1983,  84  too?) 

Fault  Location  Algorithim  Investigation 

Shipoboard  Automatic  Failure  Detection  and  Performance  Monitoring 
Artificial  Intelligence  Applications  of  ATE  (RCA  June  1983) 

Develop  Analog  ATPG  Capability 

Fault  Simulation  and  Testability  of  Electronic  Circuits  (Complete) 

Automated  Shipboard  Maintenance  (Complete) 

J.  Byron,  L.  Deight,  G.  Stratton,  RADC  TESTABILITY  NOTEBOOK  June  1982,  RADC- 
TR-82-189,  Hughes, 

Monumental  study  includes  definition  of  terms  and  their  sources 

Giordano  Associates  21  White  Deer  Plaza  Sparta  NJ  07871  Productivity  Improvement. 
Through  Testability,  1983,  $225  for  2  volumes:  one  for  management,  one  for  engineer, 

Stephen  P.  Thomas,  John  P.  Shen  "A  Survey  of:  Functional  Level  Testing  and  Testability 
Measures?*  50  pages.  Carnegie-Mellon  University  report  CMUCAD-83-18,  Nov.  1983, 
Dept,  of  Electrical  Eng. 

Discusses  logic  design  techniques  for  testability,  functional  level  test  generation,  and 
integration  of  design  and  testing  stages  with  focus  on  the  VLSI  test  bottleneck.  They 
like  Horstmann.  (IEEE  Test  Conf  83)  "Testability  Expert"  idea. 

D.  P.  Siewiorek,  R.S.  Swartz,  The  Theory  and  Practice  of  Reliable  System  Design,  1982, 
Digital  Equipment  Corp.,  772  pages. 

Very  complete  technical  book  covering  fault  avoidance,  fault  detection,  dynamic 
redundancy,  life  cycle  costs,  and  fault  statistics.  Many  examples  of  practical  reliable 
computer  system  designs  including  VAX,  IBM  system  360,  NonStop,  Bell's  ESS,  Pluribus 
(fault  tolerant  multiprocessor,  STAR(self  test  and  repair), Voyager,  SIFT,  FMPT.  It  has 
26  pages  of  references. 

Model  for  Transient  and  Permanent  Error-Detection  and  Fault-Isolation  Coverage,  D.  C. 
Bossen,  M.  Y.  Hsiao,  IBM  J  Res  Develop.  Vol.  26,  No.  1 

The  model  developed  in  this  paper  allows  the  system  designer  to  project  the  dynamic 
error-detection  and  fault-isolation  coverages  of  the  system  as  a  function  of  the  failure 
rates  of  components  and  types  and  placement  of  error  checkers,  which  has  resulted  in 
significant  improvements  to  both  detection  and  isolation  in  the  IBM  3081  Processor 

Michael  F.  Battaglia,  Angelo  A.  Bocchino,  Jr.,  Innovative  Maintenance  Utilizing  Video¬ 
disc,  IEEE  1982  AUTOTESTCON.  474-477 

This  paper  discusses  methods  of  enhancing  maintenance  testing  capabilities  by  combining 
"real  world"  video  with  expert  diagnostic  software.  Battaglia  issued  the  Navy  contract 
on  Integrated  Diagnostics  in  1984. 

Tulin  Erdim  Mangir,  Design  For  Testability:  An  Integrated  Approach  to  VLSI  Testing, 
IEEE  International  Conference  on  Computer-Aided  Design  -  1983,  September  1983,  68- 


but  "We  are  convinced  that  the  use  of  a  data  base  and  expert  system  operating  from  that 
data  base  to  be  implemented  in  a  language  such  as  PROLOG  is  the  most  expedient  way 
to  generate  alternative  strategies  for  an  integrated  testing  approach  which  is  designed 
into  VLSI  systems." 

Paul  W.  Horstman,  Design  for  Testability  Using  Logic  Programming,  1983  International 
Test  Conference  pg.  706-713.  DFT  with  expert  systems  which  can  check  for  DFT  rule 
violation. 

V.D.  Agrawal,  S.K.  Jain,  D.M.  Singen,  Automation  in  Design  for  Testability,  Custom 
Integrated  Circuits  Conf.,  May  1984,  Bell  Labs,  TITUS  =  Testability  Implementation  and 
Test  Generation  using  Scan. 

R.  G.  Bennetts,  Design  of  Testable  Logic  Circuits,  1984  164  pages  -Scan  design. 

Frederick  G.  Danner,  System  Test  Visibility-  Or  Why  Can't  You  Test  Your  Electronics? 
1983  International  Test  Conference,  635-639,  Grumman 

C.  Bellon,  Ch.  Robach,  G.  Saucier,  An  Intelligent  Assistant  for  Test  Program  Generation: 
The  "SUPERCAT"  System,  IEEE  International  Conference  on  Computer-Aided  Design  - 
1983,  September  1983,  32-33,  France 

Propose  Intelligent  Assistant  to  guide  test  designer  with  random  logic,  regular  logic. 
RAM,  PLA,  and  microprocessor  modules.  See  also  article  by  same  author  in  International 
Test  Conference  Oct  1983. 

Mark  Stefik,  Anal  Bell,  Daniel  Barrow,  Harold  Brown,  Lynn  Conway,  Christopher  Tone, 
The  Partitioning  of  Concerns  in  Digital  System  Design,  Stanford  University  HPP-82-2, 
Feburary  1982 

Frank  Tsui,  In-Situ  Testability  Design  (ISTD)-  A  New  Approach  for  Testing  High-Speed 
LSI/VLSI  Logic  Proceedings  IEEE,  Jan  1982,  59-78,  IBM 

Design  of  chips  and  modules  with  self-sufficiency  for  testability,  despite  their 
connection  in  a  system. 

J.  Jiang,  A.  Tuszynski,  Analog  Diagnostics  of  Digital  Circuits,  Proceedings;  1982  IEEE 
International  Large  Scale  Systems  Symposium  October  1982 

Nonrandom  soft  errors  can  be  quickly  found  with  analog  measurements.  This  will  be  very 
useful  for  SMART  BIT  etc. 

Gary  Engel,  A  System  Approach  to  Fault  Location,  IEEE  Fault  Tolerant  Computer 
Systems  Conference,  1983,  400-407,  Sperry-Roseville 

In-Unit-Cart-Test  (IUCT)  system  is  the  primary  diagnostic  for  Sperry  1100/90  system  and 
uses  Scan/Set  logic.  Average  fault  isolation  time  for  this  700,000  node  processor  is  6 
minutes.  The  stimulus/response  library  is  18  MBytes  and  diagnostic  is  36  MBytes. 
Estimate  that  90%+  of  hardware  faults  will  produce  valid  error  status  for  items  which 
are  like  single  stuck  at  faults. 

M.  D.  Robach,  G.  M.  Malecha,  A  Computer  Aided  Test  Analysis  System:  CATA,  Fault 
Tolerant  Computing  Symposium  1983  408-4 1 1 ,  F ranee 

Pascal  procedures:  BOARDFLOW  (3000  lines),  flow  for  equipment,  indiscernibility,  test 
path  determination  (2300  lines),  connected  by  a  database.  They  have  not  yet  added  model 
for  description  of  software  function. 


Design  for  Test  Calculus:  An  Algorithm  for  DPT  Rules  Checking,  Dilip  K.  Bhavsar,  20th 
Design  Automation  Conference.  1983,  300-307,  General  Electric-Syracuse, 

Able  to  deal  with  memory  nodes,  clocks,  etc  with  one-pass  algorithm. 

Built-In  Test/Testability  Improvement  Program  (draft),  Test  Technology  Information 
Center  Fleet  Analysis  Center,  TM-824-1658,  Dec  1982  (  August  1983),  120+ 

Z.F.  Huang,  Chen-Shang  Lin,  Ruey-Wen  Liu,  Node-Fault  Diagnosis  and  a  Design  for 
Testability,  IEEE  transactions  on  Circuits  and  Systems,  May  1983,  theory,  equations 


Ch.  Robach,  Ph.  Maiecha,  G.  Michel,  Computer  Aided  Testability  Evaluation  and  Test 
Generation,  1983  International  Test  Conference  -  Proceedings,  1983,  338-345,  IEEE 
Robach  is  with  IMAG  Laboratory  Maiecha  and  Michel  work  for  E.  P.  Schlumberger 

C.  Bellon,  Intelligent  Assistant  for  Test  Program  Generation,  IMAG  March  1984,  IEE 
#232,  page  166. 

Shunsuke  Miyamoto,  Takao  Mishida,  Tokinori  Kozawa,  Fumiya  Tada,  TRUE:  A  Fast 
Detectability  Estimation  System,  ICCAD,  1983,  Hitachi  Ltd 


— IF - - INTERMITTENT  FAILURES - 


Anton  T.  Dahbura,  Gerald  M.  Masson,  A  New  Diagnosis  Theory  as  the  Basis  of 
Intermittent-Fault/Transient-  Upset  Tolerant  System  Design,  353-356,  1982  Fault 
Tolerant  Proceedings  IEEE.  The  Johns  Hopkins  University  Grant  from  Fairchild.  Greedy 
Diagnosis  outcomes  are  used  for  all  they  are  worth.  Can  work  from  limited  information 
of  transient  faults. 

Soft  Error  Testify,  1980  IEEE  Test  Conf,  137-150,  Soft  errors  in  RAM 

Jacques  Losq,  Testing  for  Intermittent  Failures  in  Combinational  Circuits,  3rd 
USE  JAPAN  Computer  Conference,  1978,  165-170 

Is  a  study  of  the  efficiency  of  test  procedures  to  detect  intermittent  failures  in 
combination  circuits.  When  little  or  no  information  is  available,  then  the  best  method  to 
test  for  intermittent  failures  is  simply  to  repeat  the  test  designed  for  permanent 
failures.  A  simple  way  of  estimating  the  number  of  repetitions  needed  to  yield  a  high 
detection  rate  is  given. 

E.  B.  Lee,  R.  F.  Stover.,  Training  A  Logic  Network  To  Do  "Fault  Detection  In  Electronic 
Circuits,"  Curriculum  for  Test  Technology,  Nov  1983,  IEEE  83CH  1978-6,  pages  148-161. 
This  work  at  University  of  Minnesota  is  the  only  AI  type  work  on  faults  which  are  out  of 
specification  rather  than  total  failure.  Computer  programs  were  made  to  train  six 
different  pattern  recognition  schemes  with  five  decision  algorithms  on  three  simulated 
circuits  (linear  and  non-linear).  The  Arkadev  method  results  look  promising,  even 
without  the  use  of  a  computer. 


Robert  E.  Lee,  Logistical  Impacts  Within  the  Cost  Analysis  Community,  Armed  Forces 
Comptroller.  Spring  1983,  pg  18-20. 

This  information  serves  as  the  primary  LCC  cost  weights. 

B.S.  Blanchard,  Logistics  Library  Reference  List:  Suggested  Reference  List  for  the 
Logistics  Practitioner,  Logistics  Spectrum,  Fall  1983,  pg  40-45. 
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8.3  LET  OF  INTERVIEWS 

The  following  people  were  interviewed  or  contacted  in  person  or  by  telephone  during  this 
contract.  They  are  in  categories  of  Industry,  Academic  and  then  Government. 


k 


L 


Battelle,  Switzerland 

BIT  Inc.,  Manassas,  Virginia 

Boeing 

Bolt  Beranek  and  Newman 

Digital  Equipment  Corp. 

General  Electric 
General  Dynamics 
GenRad 
Grummman 
Hughes 

Lockheed  Palo  Alto  Research  Lab 
McDonnel  Douglas 
Magnavox,  Ft.  Wayne  Ind. 

RCA 

SoHar  Inc. 

Sperry 
Tektronix 
Texas  Instruments 
XEROX  PARC 


INDUSTRY 
L.  F.  Pau 
John  Cunningham 
John  Rivore  -  IDA  Study 
Bruce  Wilson  -  AI  Manager 
Dr.  Alper  Caglayan  Cambridge,  MA 

Robert  Schudy 
Hal  Shubin  -  IDT 
John  W.  Lewis,  Ph.D 
John  Hinchman,  Gordon  England 
Dr.  Geoffrey  Bunza 
Robert  Hong 

Dr.  Doug  Partridge,  Ed  Kim 
W.  Walton  Perkins 

Jim  Miller  St.  Louis,  MO 

Michael  Arington 

Eric  Braude,  Johnathen.Gaev  -  AI/ATE 
H.  Hecht  L.A.  California 

Dave  Per  sans,  Jim  Caporla 
John  Ulrich  -  had  been  DEC-IDT 
Steve  Morris  -  PEAM 

Johan  de  Kleer  Palo  Alto,  CA 


Standford  University 
MIT 

Ohio  State  University 
University  of  Illinois 
UCLA 


ACADEMIC 

Dr.  Mike  Genereseth 
Dr.  R.  Davis 

Dr.  Chandrasekaran  -  Diagnostics 
Dr.  R.  Michalski  -  Machine  Learning 
Dr.  Mangir  -  VLSI  Testability 


■  •  -  1 
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GOVERNMENT 

Wright  Patterson  AFB  AFSD  Don  Allen 

Naval  Air  Engineering  Center  N.J.  Jerry  Kunert  AI/ATE 

NRL  AI  Research  Center  Rickie  Cantone  (Now  with  Automated 

Reasoning  Corp  - 
Sunnyvale,  CA) 

Naval  Surface  Weapons  Center  William  Keiner 

Naval  Training  Equipment  Ctr.  Bob  Ahiers  Orlando,  Florida 

Human  Factors  Lab  Automatic  Knowledge 

Acquisition 


8.4  SUMMARY  OF  RELEVANT  1983  TEST  STUDY  RECOMMENDATIONS 


8.4.1  Testing  Technology 

Improving  Weapon.  System  Reliability  and  Maintainability  Study  Program),  Record 

Document  D-41,  1983.  Institute  for  Defense  Analyses. 

Tries  to  establish  requirements  for  strong  technology  program  for  DOD.  Commercial 

IR&D  efforts  have  been  directed  to  improvements  of  commercial  ATE,  not  embedded 

test.  The  services  have  installed  over  1000  different  ATE. 

Details  of  many  aspects  of  program  are  given  in  this  paper.  They  cover: 

Weapon  System  Testability  Design  Techniques 
CAD/ CAT  Design  Tools 
Logistics  Support  Activity  (LSA)  Process 
T  Prediction  and  Demonstration 
BIT/MTE/ATE/FOMS 
Fault-Tolerant  Design 
On-Line  Testing 

Performance  Monitoring  Built-In-Test  Maint  Aids 
Non-El ectronic  Design 
Off-Line  Testing 

ATE  Applications  SW  ATPG  Metrology/Calibration 
Test  Techniques 

Diagnostics/Prognostics  Advanced  Device  Testing 
Non-Conventional  T esting  System-Level  T esting 
Test  and  Evaluation 

Test  Bed  Demo  Experimental  Demonstrations 

Conclusions 

6.1  TRADITIONAL  WEAPON  SYSTEM  RELIABILITY  AND  MAINTAINABILITY  DESIGN 
TECHNIQUES  ARE  NO  LONGER  SATISFACTORY 

6.2  IMPROVEMENT  IN  THE  TECHNOLOGY  BASE  IS  REQUIRED 

6.4  THE  MANAGEMENT  OF  TESTING  TECHNOLOGY  REQUIRES  IMPROVEMENT 

Recommendations 

7.1  INITIATE  A  MAJOR  WEAPON  SYSTEM  DESIGN  TECHNOLOGY  PROGRAM, 
WHICH  INJECTS  TESTING  TECHNOLOGY  INTO  THIS  DESIGN  PROCESS 

7.2  INVEST  IN  EXPANDING  THE  TESTING  TECHNOLOGY  BASE  TO  PROVIDE  "OFF- 
THE-SHELF"  PROVEN  ALTERNATIVES  FOR  USE  IN  WEAPON  SYSTEM  DESIGN 

7.3  INSTITUTIONALIZING  THE  TRANSITIONING  AND  UTILIZATION  OF  TESTING 
TECHNOLOGY 

7.4  INITIATE  A  SERIES  OF  ACTIONS  TO  IMPROVE  TESTING  TECHNOLOGY 
MANAGEMENT 

A  summary  of  their  analysis  is  in  table  8.4.1. 
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Table  8.4~1  Test  Technology  Prioritization 


8.4.2  ARTIFICIAL  INTELLIGENCE  APPLICATIONS  TO  MAINTENANCE 


Technology  Working  Group  Report  (IDA/OSD  R&M  Study),  Anthony  Coppola,  USAF, 
Working  Group  Chairman,  Institute  for  Defense  Analyses,  IDA  Record  Document  D-28, 
August  1983. 

EXECUTIVE  SUMMARY 

The  maintenance  of  modern  military  systems  employs  a  variety  of  automation.  Built-in- 
Test  (BIT)  provides  on-line  fault  detection  and  some  isolation,  Automatic  Test  Equipment 
(ATE)  is  indispensible  at  intermediate  and  depot  repair  stations,  and  automated  mainte¬ 
nance  aids  and  trainers  abound. 

These  developments  were  designed  to  speed  maintenance  and  to  compensate  for  declining 
skill  levels  in  the  maintenance  force.  They  are  currently  far  from  satisfactory.  Modern 
maintenance  is  characterized  by  excessive  false  alarms  and  unnecessary  removals  at  all 
levels  of  maintenance. 

The  results  of  these  deficiencies  are  long  maintenance  times,  resources  wasted  in 
unnecessary  or  inefficient  maintenance  actions,  and  systems  out  of  action  which  need  not 
be.  Correcting  these  problems  would  therefore  provide  both  an  economic  advantage  and  a 
force  multiplier. 

To  create  quantum  improvements  in  maintenance  will  require  the  application  of  radical 
changes  to  the  technology.  One  possibility  is  the  application  of  Artificial  Intelligence  (AI) 
techniques  to  maintenance.  AI  is  beginning  to  see  application  to  practiced  problems  in 
many  disciplines,  and  hence  is  potentially  capable  of  relatively  rapid  implementation  into 
military  systems. 

At  present,  DoD  efforts  in  applying  AI  to  maintenance  are  small  and  exploratory. 

The  task  of  the  Artificial  Intelligence  Applications  committee  was  to  examine  the 
opportunities  for  applying  AI  to  maintenance,  assess  the  costs,  risks,  and  development 
times  required,  and  provide  recommendations  to  the  DoD  for  action. 

The  committee's  recommendations  are  detailed  in  section  9  of  this  report,  and  in  the 
attached  position  papers.  A  summary  follows: 

1.  The  DoD  should  take  advantage  of  the  relative  maturity  of  the  technology  for 
creating  expert  systems.  Specific  applications  of  maintenance  expert  systems 
should  be  started  immediately,  and  multi-application  maintenance  experts  developed 
and  standardized. 

Develop  maintenance  expert  systems  immediately  for  current  maintenance  applica¬ 
tions  where  the  existing  ATE  has  been  inadequate.  Permit  these  systems  to  be  built 
in  any  convenient  language  and  architecture,  except  that  test  programs  generated 
for  outside  use  would  be  in  ATLAS. 

Develop  versatile  maintenance  experts  for  specific  domains  (e.g.,  digital  electroncs) 
capable  of  use  in  different  systems.  System  specific  data  would  be  required  for 
each  application,  but  the  knowledge  base  would  remain  the  same. 

Develop  a  tool  to  automate  the  creation  of  the  system  specific  data  required  by  the 
maintenance  experts  described  in  the  preceding  paragraph. 


2.  Develop  "smart"  built-in-test  (BIT)  systems  to  reduce  false  alarms,  identify  inter¬ 
mittent  failures,  improve  BIT  coverage. 

3.  Fund  applied  research  in  Al  for  maintenance  to  improve  expert  system  designs  and 
to  develop  other  promising  applications.  Topics  could  include  automating  creation 
of  maintenance  manuals,  applications  to  maintenance  information  systems,  AI  based 
automatic  test  pattern  generation  (ATPG),  VHSIC  design  for  testability,  knowledge 
based  computer  aided  instruction  (CAI),  and  self-improving  diagnostics. 

4.  Foster  an  integrated  DoD-Industry  approach.  Coordinate  DoD  activity  through  a 
tri-service  working  group  under  the  existing  3LC  panel  on  automatic  testing. 
Encourage  private  avenues  of  development;  continue  to  support  industry  IR/D  in  the 
area. 


S.%  J  INTEGRATED  TESTING  AND  MAINTENANCE  TECHNOLOGIES 

Robert  O.  Denney,  Mike  3.  Partridge,  Roger  B.  Williams,  Boeing  Aerospace  Company 
Avionics  Laboratory  (AFWAL/AAAS-2)  Air  Force  Wright  Aeronautical  Laboratories, 
F33615-81-C-1517,  AFWAL-TR-S3-1183,  December  1983. 

The  Integrated  Testing  and  Maintenance  Technologies  study  effort  defined  requirements 
for  the  onboard  test  and  maintenance  system  for  the  1990's  tactical  fighter.  These 
requirements  are  documented  in  the  ITM  system  specification. 

This  report  provides  the  background  and  analyses  that  led  to  the  ITM  system  require¬ 
ments.  Problems  with  test  and  maintenance  systems  were  analyzed  and  current  tactical 
fighters  were  evaluated  to  determine  where  improvements  could  be  made.  The 
anticipated  avionic  architecture  and  mission  for  the  1990's  tactical  fighter  were 
evaluated  to  determine  new  demands  on  the  test  and  maintenance  system.  From  these, 
the  requirements  for  ITM  were  developed  by  starting  at  the  highest  level,  defining  the 
required  capabilities,  and  specifying  requirements  down  to  the  level  of  what  tests  to 
perform  and  how  the  system  operates. 

The  system  as  specified  is  primarily  a  test  data  collection  system,  with  tests  provided  by 
the  subsystem  and  test  data  processing  provided  by  the  subsystem  software.  The 
improvements  over  current  system  provided  for  in  the  specification  include— 

a.  Better  filtering  of  nuisance  alarms  from  the  pilot.  This  includes  filtering  of 
intermittent  failure  reports  when  they  do  not  affect  the  mission  and  suppression  of 
reports  of  noncritical  failures  during  critical  flight  phases. 

b.  Better  isolations  of  intermittent  faults.  This  is  accomplished  by  inflight  recording 
and  postflight  processing  of  fault  data  and  associated  data,  including  equipment 
temperature,  aircraft  flight  dynamics,  equipment  operating  modes,  and  subsystem 
modes. 

c.  Incorporation  of  more  extensive  system-level  tests.  These  include  reasonableness 
tests  of  mission  operational  data,  statistical  tests  on  the  innovation  vector  from  a 
Kalman  filter,  and  voter  tests  where  applicable. 
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d.  Variable  test  tolerances.  For  those  tests  that  are  measurements  of  performance 
against  limits  (e.g.,  power  supply  voltage,  transmitter  power)  the  limits  are  stored 
in  the  system  software  instead  of  in  the  subsystem  hardware. 

e.  Interactive  participation, by  the  maintenance  technician.  Besides  selecting  the  test 
sequences  and  observing  results,  the  maintenance  technician  receives  test  data  and 
the  diagnostic  decisions  the  system  used  in  isolating  a  problem.  These  are  used  by 
the  maintenance  technician  along  with  the  capability  to  access  additional  test  data 
to  isolate  the  difficult  problems.  The  interaction  with  the  system  provides  "on  the 
job  training"  for  the  maintenance  technician  instead  of  relegating  him  to  the  role 
of  a  "button  pusher." 

f.  Offline  use  of  test  data.  Test  data  that  will  help  the  depot  to  isolate  failures  or 
determine  the  cause  of  them  is  stored  for  access  at  the  depot.  This  data  can  also 
be  used  to  create  a  failure  history  to  develop  failure  trends. 

Another  conclusion  of  the  ITM  effort  is  that  artificial  intelligence,  expert  systems,  is 
potentially  useful  for  application  to  testing  and  maintenance  problems  and  appears 
practical  for  postflight  analysis  applications  in  future  generation  tactical  fighter 
aircraft.  The  benefit  of  an  expert  system  is  that  it  can  help  solve  problems  normally 
requiring  the  experience  of  the  system  designer  or  an  experienced  maintenance  techni¬ 
cian. 

While  the  use  of  expert  systems  appears  beneficial  and  practical,  it  still  requires  more 
development  work  before  it  can  be  applied  on  a  tactical  fighter  development  contract 
for  integration  with  its  avionic  system.  No  systems  with  AI  have  yet  been  developed  for 
avionics,  and  most  miltary  applications  are  just  now  getting  started.  This  makes 
specification  and  management  of  its  development  for  a  project  difficult  and  risky. 
Continued  development  of  the  expert  system  described  in  this  study  or  a  similar 
development  effort  on  an  RicD  basis  is  recommended  as  a  risk  reduction  measure  before 
insertion  in  a  mainstream  program. 


8.4.4  COMPUTER  AIDED  TESTABILITY  DESIGN  ANALYSIS 


E.  Shemeta,  R. Spillman,  Boeing  Dec  1983,  RADC  TR-83-257.  Condensed  as  paper 
authored  by  Thorns  L.  Fennell,  Thomas  A.  Nicolino,  Reliability  and  Maintainability 
Symposium  1984,  pages  1-6. 


Summary  of  Results 


A  design  plan  for  a  CAT  system  has  been  developed  that  naturally  fits  into  the  flow  of 
the  engineering  design  process  and  is  integratable  with  existing  CAD  systems.  This 
design  plan  requires  the  development  (or  refinements)  of  five  CAT  modules;  these 
modules  are: 

1.  Testability  Allocation  Module 

2.  Test  Point  Location  Module 

3.  Testability  Analysis  Module 

4.  BIT  Selector  Module 

5.  Automatic  Test  Program  Generation  Module 

These  CAT  modules,  when  integrated  into  existing  CAD  systems,  will  provide  a  designer 
with  an  interactive  design  tool  to  ensure  that  testability  is  incorporated  into  the  design 
process. 

A  key  to  an  effective  CAD/CAT  system  is  the  utilization  of  detailed  design  information 
that  typically  exists  in  the  databases  of  today's  commercial  CAD  systems.  Much  of  the 
design  information  in  that  database  is  useable  for  CAT  design  tasks;  although,  when  CAT 
functions  are  added,  this  database  must  be  expanded  to  include  additional  information 
required  in  the  design  for  testability  process.  The  structure  of  the  existing  databases 
and  their  management  systems  are  adequate  to  support  this  expansion.  For  example, 
standard  designs  of  commony  used  logic  or  functional  structures  are  saved  in  the  CAD 
database  for  use  by  other  designers.  Functional  structures  with  proven,  effective,  BIT, 
can  also  be  stored,  along  with  appropriate  notational  information,  in  the  same  CAD 
database  to  provide  useful  inputs  to  the  testability  programs.  Testability  unique 
functions,  such  as  BIT  selection,  may  require  separate  files,  but  there  should  be  no 
problem  in  using  the  CAD  system's  file  and  data  base  handling  software  to  service  those 
needs. 

Currently  no  commercially  available  CAD  system  performs  all  these  functions  on  an 
integrated  basis.  A  number  of  CAD  systems  have  been  developed  for  semiconductor 
LSI/VLSI  chip  design.  Some  automatically  incorporate  BIT,  such  as  Level  Sensitive  Scan 
Design  (LSSD),  into  a  design.  Such  systems  are  not  operational  at  the  Printed  Circuit 
Board  (PCB)  or  unit  level  design,  due  to  limitations  of  currently  available  parts.  For 
example,  the  testability  of  a  microprocessor  based  design  is  dependent  upon  the 
firmware  and  software  as  well  as  the  hardware  interconnections.  However,  for  a  very 
broad  spectrum  of  design,  the  CAD/CAT  design  plan  described  here  is  practical,  simple 
to  implement,  and  an  improvement  over  existing  manual  methods. 

The  use  of  engineering  workstations,  with  continuation  of  schematic  based  CAD  tools,  is 
now  emerging  as  the  primary  design  tool  for  the  electronics  engineer.  The  CAT 
approaches  described  in  this  document  are  compatible  with  this  trend.  Some  of  the  CAT 


modules  needed  already  exist  or  have  been  developed  to  a  point  that  a  practical 
production  version  could  be  developed  economically.  For  instance,  many  ATPG  programs 
are  available  and  in  use  in  present  CAD  systems;  testability  analysis  approaches  have  been 
developed  by  Grumman  (non-automated  for  PCB's  only),  Boeing  (automated  system 
testability)  and  Comsat  General  Corporation  (a  commercial  add-on  to  TEGAS).  Further 
details  of  the  recommended  CAT  approach  are  presented  in  section  3.0,  CAT  Design 
Plan,  and  section  5.0,  CAT  Guidelines. 
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9.0  GLOSSARY  OP  TESTABILITY  TERMS 


The  sources  of  some  definitions  are  given  in  parentheses  following  the  definition.  The 
source  identification  are  from  the  MATE  document: 


ATG 

IEEE 

IEEE/FTC 

I/JS 

TR-3826 


Industry  ATG  Glossary,  Report  of  Industry  Ad  Hoc  ATE  Project  for  the 
Navy,  April  1977 

IEEE  Standard  Dictionary  of  Electrical  and  Electronics  Terms,  IEEE  Std 
100-1972 

Interim  IEEE  Technical  Committee  on  Fault  Tolerant  Computing  Dic¬ 
tionary  of  Terms. 

Industry/3oint  Services  Automatic  Test  Project  Final  Report 
A  Framework  for  Designing  Testability  into  Electronic  Systems 


MIL-STD-721B  Definition  of  Effectiveness  Terms  for  Reliability,  Maintainability 
Human  Factors  and  Safety 


MIL-STD-1309B  Definitions  of  Terms  for  Test,  Measurement  and  Diagnostic  Equipment 


MIL-STD-2077  Test  Program  Sets,  General  Requirements  for 

Ambiguity  group.  The  group  of  maintenance  replaceable  units  which  may  contain  faults 
which  result  in  the  same  fault  signature;  also  the  number  of  units  in  such  a  group 
(TR-3826). 


Automatic  test  equipment  (ATE).  Equipment  that  is  designed  to  conduct  analysis  of 
functional  or  static  parameters  to  evaluate  the  degree  of  performance  degradation  and 
that  may  be  designed  to  perform  fault  isolation  of  unit  malfunctions.  The  decision 
making,  control,  or  evaluative  functions  are  conducted  with  a  minimum  reliance  upon 
human  intervention  (MIL-STD-1309B). 


Availability.  A  measure  of  the  degree  to  which  an  item  is  in  the  operable  and 
committable  state  at  the  start  of  the  mission,  when  the  mission  is  called  for  at  an 
unknown  (random)  point  in  time  (MIL-STD-721B).  Availability  is  the  probability  of  a 
system  readiness  over  along  interval  of  time  (TR3826). 

BIT  false  alarms.  A  false  alarm  occurs  if  a  component  is  declared  defective  and  it  is 
found  in  separate  off-line  test  to  be  failure  free. 

Built-in  test  (BIT).  A  test  approach  using  BITE  or  self-test  hardware  or  software  to  test 
all  or  part  of  the  unit  under  test  (UUT)  (MIL-STD-1309B). 

Built-in  test  equipment  (BITE).  Any  device  which  is  part  of  an  equipment  or  system  and 
is  used  for  the  express  purpose  of  testing  that  equipment  or  system.  BITE  is  an 
identifiable  unit  of  the  equipment  or  system  (MIL-STD-1309B). 

Dependent  fault/dependent  failure.  A  fault  which  is  caused  by  the  failure  of  an 
associated  item  (MIL-STD-721B). 


Design  fault.  A  design  characteristic  of  either  hardware  or  software  which  causes  or 
materially  contributes  to  equipment  malfunction  independent  of  the  presence  of  hard¬ 
ware  failures  (TR3826). 

Design  for  testability  (DFT).  A  design  process  or  characteristic  thereof  such  that 
deliberate  effort  is  expended  to  assure  that  a  product  may  be  thoroughly  tested  with 
minimum  effort,  and  that  high  confidence  may  be  ascribed  to  test  results  (TR3826). 

False  alarm.  An  indicated  fault  where  no  fault  exists.  (Does  not  include  good  items  in 
an  ambiguity  group.)  (MIL-STD-1309B). 

Fault.  A  physical  condition  that  causes  a  device,  component,  or  element  to  fail  to 
perform  in  a  required  manner;  for  example,  a  short-circuit  or  a  broken  wire  (IEEE).  A 
degradation  in  performance  due  to  detuning,  maladjustment,  misalignment,  failure  of 
parts,  and  so  forth  (MIL-STD-1309B).  The  causative  failure  of  a  lower  level  assembly 
within  the  unit  under  test  (ultimately  a  fault  may  be  traced  to  a  physical  change  of  a 
component  of  the  system  )  (I/JS). 

Fault  coverage.  An  attribute  of  a  test  or  test  procedure  expressed  as  the  percent  of 
faults  of  the  total  fault  population  which  that  test  or  test  procedure  will  detect 
(TR3826). 

Fault  detection.  A  process  which  discovers  or  is  designed  to  discover  the  existence  of 
faults;  the  act  of  discovering  existence  of  a  fault  (TR3826).  One  or  more  tests 
performed  to  determine  if  any  malfunctions  or  faults  are  present  in  a  unit  (MIL-STD- 
1309B). 

Fault  isolation.  Where  a  fault  is  known  to  exist,  a  process  which  identifies  or  is  designed 
to  identify  the  location  of  that  fault  within  a  small  number  of  replaceable  units 
(TR3826).  Tests  performed  to  isolate  faults  within  the  unit  under  test  (MIL-STD- 1 309B). 

Observability.  An  attribute  of  equipment  design  which  describes  the  extent  to  which 
signals  of  interest  may  be  observed  (TR-3826). 

On-line  test.  Test  of  a  UUT  in  its  operational  environment  (MIL-STD- 1309B). 

On-line  testing.  Testing  of  the  unit  under  test  in  its  operational  environment  (see 
interference  testing  and  noninterference  testing  (MIL-STD-1309B). 

On-line  test  equipment.  Equipment  used  to  perform  tests  on  a  UUT  while  the  unit  is  in 
its  normal  operating  environment  (ATG). 

Readiness.  A  state  of  being  ready  to  successfully  perform  or  being  in  the  act  of 
successfully  performing  a  defined  mission  (TR-3826). 

Readiness  test.  A  test  specifically  designed  to  determine  whether  an  equipment  or 
system  is  operationally  suitable  for  a  mission  (MIL-STD- 1309B). 

Self-test.  Built-in  test  (TR3826).  A  test  or  series  of  tests,  performed  by  a  device  upon 
itself,  which  shows  whether  or  not  it  is  operating  within  designed  limits.  This  includes 
test  programs  on  computers  and  automatic  test  equipment  which  check  out  their 
performance  status  and  readiness  (MIL-STD- 1309B). 


Self-test  capability.  The  ability  of  a  device  to  check  its  own  circuitry  and  operation. 
The  degree  of  self-test  is  dependent  on  the  ability  to  fault  detect  and  isolate  (MIL-STD- 
1309B). 

SIT  (System  Integration  Test).  Built-in  system  test  that  is  centrally  integrated,  i.e.,  BIT 
data  is  analyzed  through  a  central  computer  before  Fault  Detection/Isolation  can  be 
determined. 


Testability.  A  design  characteristic  which  allows  the  status  (operable,  inoperable,  or 
degraded)  of  a  system  or  any  of  its  subsystems  to  be  confidently  determined  in  a  timely 
fashion  (TR-3826).  Testability  attempts  to  quantify  those  attributes  of  avionic  system 
designs  which  facilitate  detection  and  isolation  of  faults  that  affect  system  perform¬ 
ance.  Testability  has  been  defined  as  "the  characteristic  of  a  design  which  allows  the 
status  of  a  system  or  any  of  its  subsystems  to  be  confidently  determined  in  a  timely 
fashion."  This  definition  should  be  expanded  to  include  the  concept  of  isolating  and 
repairing  detected  faults  in  a  confident  and  timely  fashion  so  as  to  minimize  repair  time. 
Testability  is  the  unambiguous  isolation  of  a  fault  to  an  appropriate  level  of  replaceable 
equipment  such  that  the  proper  maintenance  is  implemented  with  a  minimum  expendi¬ 


ture  of  time  and  resources.  Specification  of  that  appropriate  level  is  a  most  important 
consideration,  which  must  be  addressed  early  in  the  design  cycle.  (Subtask  A-3  Design 
for  Testability,  Final  Report.  U.  S.  Government  Printing  Office,  1977,  727-156/1-3, 
P  35.) 

Testability  reflects  the  susceptibility  of  a  PCB  to  the  detection  of  all  faults,  to  rapid  and 
accurate  isolation  to  the  faulty  component  without  ambiguity  and  to  functional  test  and 
thereby  verification  of  PCB  performance  as  specified  and/or  required,  (testability 
Investigation  Attachment  IV  to  Interim  Report,  Manufacturing  Methods  and  Technology 
for  Digital  Fault  Isolation  for  Printed  Circuit  Boards,  Contract 
No.  DAAK40-78-C-0290.) 

Testability,  inherent.  A  hardware  testability  design  characteristic  which  does  not 
include  consideration  of  test  stimulus/response  data. 

Testability  Relative  Overhead  (Burden).  The  incremental  increase  in  the  prime  func- 
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9.1  LIST  OF  ABBREVIATIONS  AND  ACRONYMS 


AI 

AIAM 

AR1NC 

ATE 

ATLAS 

ATPG 

BIT 

BITE 

CAPDT 

CND 

CRIB 

CSS 

C3I 

DEC 

DFT 

EEPROM 

ETE 

FA 

FIT 

FOM 

GE 

H/W 

IBM 

IC 

IDA 

IDEAL 

ITM 

LCC 

LISP 

LRU 

LSSD 

MATE 

MTBF 

PROLOG 

RAM 

RADC 

RTOK 

SID 

SIT 

STAMP 

S/W 

SPR 

SRU 

STE 

TPS 

VAV 

VAX 

VHSIC 

VLSI 


artificial  intelligence 

artificial  intelligence  applications  to  maintenance  (see  8.4.2) 

Aeronautical  Radio  Inc. 

automatic  test  equipment 

Abbreviation  Test  Language  for  All  Systems 

Automatic  Test  Program  Generation 

built-in  test 

built-in  test  equipment 

computer-aided  preliminary  design  for  testability 
cannot  duplicate 

computer  retrieval  incidence  bank 
consolidated  support  system 

command,  control,  communication,  and  intelligence 
Digital  Equipment  Corp. 

Design  for  Testability 

electrically  eraseably  programable  read  only  memory 

electronic  test  equipment 

false  alarm 

fault  isolation  test 

figure  of  merit 

General  Electric 

Hardware 

International  Business  Machines 

Integrated  Circuit 

Institute  for  Defense  Analysis 

Incorporation  of  Diagnostic  Experience  through  Automated  Learning 

Integrated  Testing  and  Maintenance 

life  cycle  cost 

an  AI language 

line  replaceable  unit 

Level  Sensitive  Scan  Design 

modular  automatic  test  equipment  -  Air  Force 

mean  time  between  failures 

an  AI language 

reliability  A  maintainability 

Rome  Air  Development  Center  (USAF) 

retest  OK 

Self-Improving  Diagnostics 

System  Integrated  Test 

System  Testability  and  Maintenance  Program 

Software 

Software  Problem  Report 
shop  replaceable  unit 
special  test  equipment 

Test  Program  Set  (ATE  Hardware  and  Software) 
verification  A  validation 
computer  manufactured  by  DEC 
Very  High  Speed  Integrated  Circuit 
Very  Large  Scale  Integration 
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